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FOREWORD 



Following nearly a decade of education reforms aimed at raising the performance 
of/ American high school students in academic subjects, there has emerged a broad 
consensus for improving the quality of vocational programs as welL Because so many 
students tal;e vocational courses,.regardless of their career plans, and because secondary 
school, vocational training can affect job productivity and performance, the content and 
management of vocational programs warrant careful attention. 

To assist Congress in reauthorization of the Carl D. Perkins Vocational Education 
Act, the House Committee on Education and Labor asked OTA to explore the possibility 
of instituting specific requirements for performance measurement of the Nation's 
secondary school vocational programs. OTA examined the technical feasibility and 
utility of performance stimdards based on labor market indicators and scores on 
occupational competency tests, and looked at other possible measures as well. 

This Background Paper contains the results of OTA's analysis. Throughout the 
study OTA sought and received the assistance of many individuals arid organizations in 
the business^ education, and .government communities. Their thoughtful contributions 
and, criticisms were invaluable, although their participation does not necessarily reflect 
their endorsement of the contents of the report, for which OTA bears sole responsibility. 
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Performance Standards 
For Secondary School Vocational Education 

SUMMARY 

To assis": Congress in preparing for the reauthorization of the Carl D. Perkins 
Vocational Education Act, OTA was asked to examine the technical feasibility and utility 
of instituting specific requirements for performance measurement of secondary school 
vocational programs; The Perkins Act currently specifies the composition of State 
advisory boards^ which are\eharged with developing skiUs inventories, the establishment 
of evaluation criteria, and biennial program evaluation. The act also mandates that 
States assess the quality of vocational programs in terms of workplace requirements and 
occupational preparation of students. But these requirements are not rigorously 
enforced. The education and business communities now increasingly support the view 
that more exacting measurement of the quality of high school vocational programs may 
be needed. OTA examined two types of performance measures, but did not evaluate the 
overall strengths and weaknesses of a Federal decision to mandate these or other 
measures. 

CONTEXT 

The secondary school vocational system is complex. Recent data indicate that 
virtually all American high school students take at least one vocational course^ one-half 
of all students take four or more vocational coursas, and there is almost no difference in 
the number of vocational course credits taken by students in different racial and ethnic 
groups. In addition, almost one-half of aU vocational credits are taken by college-bound 
students, and there is growing recognition that aU students — vocational or not — need 
academic skills to function productively when they leave high school. 



' Interest in measurement of vocational program quElity comes at a time of 
heightened concern for the relationship between the Nation's educational system and the 
future of the American economy. Growing evidence of a work force ill-'^quipped for 
many jobs that require higher skill levels, demand by industry for workers able to learn 
hew skills and adapt to new technologies, concern for the economic well-being of 
approximately 20 million: noncollege-bound youth, and the recognition that many 
vocational students who go on to college are well served b^ job skills has spurred 
education ,and business leaders to redefine the objectives of secondary school vocational 
programs and to demand improvements in their quality. There is now widespread 
consensus for including the vocational education system in the national debate over 
school reform and academic excellence. 

OUTCOME MEASURES 

One manifestation of the ccncern for quality of vocational programs has been the 
interest in using outcome measures as indicators of program effectiveness. The 
application of outcome-based performance standards in other Federal employment and 
training programs, such as the Job Training Partnership Act, has led many observers to 
call for a similar strategy in vocational education. 

OTA looked closely at two outcome measures: 1) indicators of labor market 
performance of vocational graduates, including job placement, earnings, and duration of 
unemployment; and 2) competency in occupationall^ -relevant skills, as measured by 
scores on tests of vocational ability. These are the most frequently discussed measures, 
are currently the most quantifiable, and — most important they reflect the broadly 
accepted definition of the principal objectives of secondary level vocational training: 
the preparation of high school students for productive and gainful employment. 
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There are a number of other outcomes that could be included in assessments of 
program quality* OTA discovered considerable interest in widening the scope of 
performance meabiiremeht to account for the multiple objectives of secondary vocational 
training. While OTA did iot analyze these measures in detail, they do warrant continued 
attenti%»n« For example, many observers have suggested that vocational programs may 
reduce the high school dropout rate, and that keeping students in school who might 
otherwise quit could be counted as a positive outcome of vocational programs* Similarly, 
participation of vocational graduates in.postsecondary education could be counted as a 
positive outcome* Other measures, such as the merket value of goods produced by 
vocational students while still in school (e.g.9 in cooperative education programs) and the 
likelihood bf vocational graduates being selected for employer-provided training, have 
also received attention. 

OTA also learned that, many States, in response to growing pressure for academic 
reform, have begun to introduce academic material into the vocational curriculum and to 
try to teach so-called "higher-av'der thinking skills" that will benefit students throughout 
their careers. Many people now wish to see academic achievement includes in vocational 
performance measurement, and would like to see im£: movements in testing technology to 
provide better assessments of higher-order cognitive abilities. 

Because measurement usually implies the development of standards that ehn 
become the basis for sanctions or incentives, selection of any outcome measures will 
have a strong impact on program management an^ curriculum design. Thus, as Congress 
debates the feasibility and utility of performance measurement, it will undoubtedly 
engage in a broader discussion of the objectives of the vocational education system. 
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PRINCIPAL FINDINGS 

Labor Market Indicators 

Job place mentSi wage ratesi earningSi and duration of employment ani 
unemployment of students who complete defined courses of vocational study can provide 
important clues to pro-am quality. If graduates of two programs with similar objectives 
and in the same or similar communities experience significantly different labor market 
outcomes^ the relative quility of the two programs can be said to differ. Indeed, because 
a primary objective of vocational training is productive employmenti how vocational 
program graduates fare in the labor market can be an important reflection of the 
programs from which they graduated. 

But technical and methodological problems have always cr€ated barriers to 
effective use of labor market indicators for program evaluation. These problems include 
the questionable validity of information provided by program graduates about their 
current employment and prior coursework, the potential for bias in data provided by 
school personnel (especially if the performance standards become a oasis for program 
funding)) and the high costs of conducting folio wup surveys of a mobile labor force. 

Recent improvements in the quality and cost of data coUection and storage, made 
possible by mandated changes in employer repo^-ting of wage and earnings data, provide 
partial relief for these concerns and increase the feasibility of designing cost-effective 
labor market outcome indicators. In particular, the use of wage records maintained by 
the States in compliance with recent amendments to the Social Security Act cculd be a 
first step toward improved labor market indicato:rs of program performance. The wage 
records datD are more accurate than self-renorted survey information, allow for longer- 
term evaluations of employment, and can be merged with other data such as military 
records or computerized school transcripts. 
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There remain, some important technical issues to be resolved before the wage 
recor.is system could be implemented for secondary vocational program evaluation. 
FirstV school transcript information js not uniform, despite recent efforts to standardize 
vocational course definitions aiid curricular offerings. In addition, clarification of 
confidentiality restrictions pertaining to individual financial daxa> and the decision to 
maintain long-term archives of earnings and employment data, would be minimal 
prerequisites to implementation of the wage records system for performance evaluation. 

Even with these technical improvements, however, labor market indicators alone 
are an insufficient basis for performance standards, for several reasons: 

It will always be difficult to isolate the specific etfscts of school 
programs from the geographical, demographic, ana '^cher nonschool 
factors that determine individual success in the labor market, and it is 
prohibitively costly to collect and analyze the detailed student 
background data that would be necessary to overcome this problem. 
Alternative solutions, such as conducting controlled experiments or 
using sophisticated statistical correction methodologies, are not 
practical. 

• The use of some labor market indicators could distort school 
behavior: placement rates, for example, especially if used as the basis 
for sanctions or rewards to schools, could induce schools to 
concentrate toe heavily on coaching students in job search and 
interview skills at the expense of teaching vocational competencies 
that would benefit them in the workplace. 



• - « IdeaV labor market ind!catorsr would include information on items not 
currently pax*t of the wage records system^ such as participation in 
employer-sponsored trainings measures of employee morale^ job 
performance, and productivity. These types of information are 
difficult and costly to obtain* 

For these reasons, OTA finds that labor market indicators are an important but 
insufficient measure of overall school or program performance. 

Tests of Acquired Skills 

Competency tests designed to assess mastery of skills can also be valuable as part 
of a comprehensive approach to program evaluation and can provide useful feedback to 
local program staff. When linked to data reflecting labor market needs, the tests are 
useful for evaluating the relevance of vocational curricula to current and anticipated 
conditions. In addition, tests that measure more broadly defined developed abilities can, 
when used properly, contribute relevant information about the types of courses offered 
by schools and about efforts to provide special guidance or coursework to students who 
need it most. For performance measurement, tests designed expressly for system-wide 
program evaluation are preferable to those meant for assessment of individual abilities. 

The States have become very active in the development and use of occupational 
competency tests. Detailed paper-and-pencil tests, as well as hands-on tests of technical 
proficiency — which are often designed with the cooperation of experienced workers — 
can provide valuable information about the quality of vocational programs. State-of-the- 
art tests that rely on computer-based interactive systems, which are currently being 
designed, may provide important technological improvements over conventional testing 
modes. 

14 
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Occupational competency tests, like labor market outcomes, provide important but 
insufficient indications of program effectiveness, and could not become the sole basis for 
perfprmance standards, for several reasons: 

• It is never possible to know exactly how much of an individual's " 
developed abilities can be attributed to a school or program. Accurate 
interpretation of test results requires a high level of sophistication in 
accounting for socioeconomic and other correlates of test 
performance. 

• Depending on the type of test used (e.g., multiple choice) there is a 
risk that schools will coach students on test-taking strategies and on 
specific test items at the expense of teaching the. skills and concepts 
purportedly measured by the tests. 

• Most tests are designed to measure the upper limit of what a person 
can do, and do riot necessarily indicate how a person would typically 
perform at work. 

• There are too many different occupations taught in American schools 
for there to be a viable nationc^ competency test. In addition, tests 
that accurately assess what is taught in specific courses could 
encourage excessive emphasis on nighly specialized skills at the 
expense of more broadly applicable generic skills. There is a growing 

" consensus . that the pace of workplace innovations will require 
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flexibility: for example, many people have argued that vocational 
graduates- need 'learning-tt -learn skills" and the ability to work 
effectively in teams, as well as job*specific proficiencies. 

The Federal Role 

Performance standards based on labor market indicators or test scores are useful, 
but neither approach can fully capture the complex goals of secondary vocational 
education* An important role for the Federal Government to play is to encourage a 
broad vieyr toward performance measurement. 

In addition, the design of standards depends on the definition of vocational program 
objectives. Because both the definition and attainment of program objectives are 
sensitive to local and regional differences, there is no single measure that can yield a 
national stazidard. The Federal Government could effectively encourage. and support the 
States, individually or in consortia, in their current efforts to define outcomes and devise 
appropriate ni^^asures of performance. In particular. Congress could play a leadership 
role by helping States 

• develop clear definitions of the objectives of their vocational 
programs; 

• establish agreed-upon norms of measurement and guidelines for data 
c^Uection and analysis; 

• carry out research aimed at the development of improved technologies 
for testing; 
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conduet;pilot;demohstPations of the f-fiects of alternative performance 
standards.cn school and student behavior; and 

raise necessary funds for dissemination of innovations in performance 
assessment -methodologies. 
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Chapter! 
Introductiofl 

AN ATMOSPHERE OF REFORM 

Rapid technological change in the workplace, coupled with intense international 
competition, have focused national attention once again on the role of the schools in 
preparing youth ior productive employment. The ominous news seems relentless: 
declining test scores, growing numbers of high school students unable to perform simple 
arithmetic, poor showings of American youth in international comparisons of academic 
ability, a national dropout rate of roughly 25 percent, and forecasts of relatively high 
demand for workers with technical skills. There are many reasons to wish for better 
educational opportunities for American school children; But economic considerations — 
the^>roduct2vity growth slowdown and America^ struggle to remain competitive in the 
global economy — have provided the main impetus for reform. And while it is erroneous 
to pin the blame for America's economic difficulties entirely on the education system, 
many people believe that basic features of the school system — who is taught which 
subjects? when? how? — need to be redesigned to fit the realities of the post-industrial 
world. ^ 

Things are changing. As OTA pointed out in a -recent report on educational 
technologies, American schoolrooms today still resemble their ancestors of 50 years ago 
more closely than do business organizations, manufacturing plants, hospitals, or 
university research facilities. But there is an extraordinary consensus in the country 

1. It is important for school reformers to avoid mistaking the coincidence of test 
score, decline and economic downturn for cause and effect. See Richard Murnane, 
"Education and the Productivity of the Work Force: Looking Ahead," American Living 
Standard^f Robert E. Litan et ah, (eds.) (Washington, DC: Brookings Institution, 1988), p. 
215; and John Bistjp, "Why the Apathy in American High Schools?" Educational 
Researchei, yqU 18, No. 1, January-February 1989, pp. 6-iO. 

2. U.S. Cpngress, Office of Technology Assessment, Power On! Ne^v Tools for 



today that teQhnological innovation at the workplace, changing demogr^iphic composition 

of - the school population, modern understanding of multiple lea>*:iing styles, and the 

advent of new learning technologies need to be reflected in the way schools are 

organized and in the way instruction is designed. At least since 1983, when the 

Department. of Education declared this was a "Nation at Risk,"'' nearly all the States 

have begun implementing reforms: increased graduation requirements, more 

standardized testing, lengthening the school day and the school year, imposing sanctions 

for poor performance, and designing new teacher certification processes are the most 
4 

comncon strategies. 

While there is much variation in the way the States have approached the design and 
management of change, overall "accountability" iias become a guiding principle. Schools 
are expected to act like businesses, and account for their successes and failures; and 
though it has never been very easy to apply quantitative measures to complex 
educational processes and outcomes, more and more school systems have been doing just 
that. In many places, performance measurement has become a basis for incentives or 
sanctions directed at various levels of the educational system. For example, merit pay 
for teachers is an attempt to reward teachers for superior performance, while so-called 
"bankruptcy" laws penalize school districts for poor performance by transferring their 
assets and liabilities.^ 



Teaching and Learning, OTA-SET-379 (Washington, DC: U.S. Government Printing 
Office, September 1988); 

3. U.S. Department. of Education, A Nation at Risk: The Imperative for Educational 
Reform (Washington, DC: U.S. Government Printing Office, April 1983). 

4. For different views on the potential for education reform, see Thomas B. Timar and 
David L. Kirp, "State Efforts to Reform Schools: Treading Between a Regulatory Swamp 
and an English Garden," Educational Evaluation and Policy Analysis^ vol. 10, No. 2, 
summer 1988, pp. 75-88; John Chubb, "Why the Current Wave of School Reform Will 
Fail," The Public Interest^ No. 90, winter 1988, pp. 28-49; and Chester Finn, 
"Questioning Cliches of Education Reform,'* Education Weekf Jan. 25, 1989, p. 40. 

b. For a review of incentive and sanction programs instituted in various places in the 
United States,^ see. David Stern, University of California, Berkeley, "Performance 
Incentives for Secondary Vocational Education," OTA contractor report, December 1989. 
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.TBDE ROLE OF VOCATIONAL EDUCATION 



Most high school performance measures of the past decade have emphasized 
academic test scores and performance in college* Occupational! y-^pecific learning has 
received less attention, and the vocational education system has remained at the fringes 
of the major reform debates. Because.of the perception that effective job training at 
the secondary level is an important element of economic resurgence, the education and 
business communities now agree that the time has come to position the future role and 
structure of vocational education squarely in.the broader education policy debate. 

Vocational education is a complex system. With its origins dating to early 20th 
century demand for skilled workers, vocational education has been traditionally viewed 
as the principal training ground for noncollegerbound youth entering technical trades. 
But recent data indicate that enrollment in high school vocational education is nearly 
universal: 97 percent of the high school graduates from the class of 1982 enrolled in at 
least one vocational course, one-^half of all students took at least four vocational courses, 
and most students who took one vocational course followed through with a second or 
later course in a sequence leading to specific occupational proficiencies. 

Most stereotypes about vocational education are inaccurate. For example, there is 
almost ho difference in the number of vocational course credits taken by white, Black, 
Hispanic, and Asian students. As shown in figure 1-1, the range is from about three 
credits, on average, for Asian students to slightly over four credits among Bl^oks. 
Perhaps more striking are the data illustrated in figure 1-2: the percentage of students 
taking two vocational course credits is roughly the same whether or not they plan to 
attend college; and roughly 10 percent of students planning to go on to graduate or 



6 U.S. Department of Education, First Interim Report From the National Assessment 

of Vocational Education (Wiashington, DC: January 1988), p. 1-5. 
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Figure 1-1 



Vocational Education Credits Taken in Higli School, 
by Race of Students 




SOURCE: Office of Technology Assessment, based on the Notional Assessment of Vocational 
Education. FIrat Inl^Hm R<pbri (Wflshlngton DC: U.S. Deportment of Education, January 1988). 
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Figure 1-2 
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Completion of VocoHonal Course Credits in 
Primary Subjects, by Students' Postsecondary Plans 



students taking no 
vocational credits 



Students taking two 
vocational credits 



Students taking four or 
more vocational credits 




None Votech'^ Some college BA/BS Grad/professional 

Postsecondary plans 



Does not include percentages of students taking one vocationa! 
course credit or three vocational course credits. Columns do not 
total 1 00 percent. 
' Vocational/Technical colleges. 



SOURCE: Offfct of Ttchnology Asstssrntnt. bastd on th« Naltonat Assessmtnt of Vocational 
Education, Rrat Interim Rtpori (Washington DC: U.S. Deportment of EducoUon. January 1988). 
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s professional school take four or more credits in vocational courses* In addition, almost 
cne-half of all vocational credits are taken by college-bound students/ 

Figure 1-3 shows that ability of vocational students, as measured by verbal and 
mathematics achievement tests, is also a poor predictor of participation in vocational 
courses.v Students ranking in the middle two ability quartiles took an average of just 
under five credits and accounted^ for more than 50 percent of vocational enrollment. 
Oniy in the highest 'Tility quartile did participation decline significantly. Socioeconomic 
status had much the same effect, as illustrated in figur -4. 

Outcome Measures for Secondary Vocational Education 

The Carl D. Perkins Vocational Education Act reflects traditional congressional 
concern with equity in access to public education. Pof example, it mandates procedures 
to be used b^ the States in allocating portions of set-aside grants for handicapped and 
economically or educationally disadvantaged students. But the Perkins Act also 
recognizes the potential rol^ of outcome measurement in vocational programs: the act 
requires States to establish boards to review vocational curricula and their relevance to 
labor market needs, and specifies that the boards must be composed of business and 
education leaders in the community according to Federal guidelines. However, these 
outcome requirem(>nts do not include methodological guidelines, and they are not 
rigorously enforced. 

The reauthorization of the Perkins Act comes at a time when performance- 
incentives and quality indicators are very much in vogue, in the schools as well as in job 
training programs outside the schools (see box 1, page 21). Many people are urging 
Congress to apply similar measures for secondary vocational education. As one 
prominent researcher explained: 

7. See John G. Wirt, "National Assessment of Vocational Education," testimony before 
the.House Committee on Education and Labor, Subcommittee on Elementary, Secondary, 
and Vocational Education, Mar. 7, 1989, p. 6. 
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Figure 1-3 



Student Ability^ and Enrollment 
in Vocational Education 
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SOURCE: Offkt of Tr.hnot' >y Asstssmtnt. bostd on Iht National A33«s3m<nt of VocaKonat 
IducaWon^ Rfst Ms'lm RgpoH (WQ>h!ng<rtn DC: U.S. DtpaHmtnl of Education. January 1988). 
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Figure 1-4 



Students' Family Income and Participation 
in Vocational Education 
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"Vocational education is. a good candidate for performance-based « • . 
policies fop several reaspns. Firstji [it] has a long though somewhat ill- 
defined tradition of jdefining accountiEtbility in terms of outcomes. For many 
yeai^ Federal vocational education policy has urged that the effectiveness 
of vocational^rograms be measured in terms of such labor market outcomes 
as plAcentcni in jobs related to training and empipyer satisfaction^ • . ; 
Secondf vocatipnsd education has made extensive use of competency-based 
curricula and coinpetehcy testing, which leiid themselves to establishing 
more performance-oriented public policies. • ^ Third^ experience with 
performance standanb and other outcome-based features of the Job 
Training Partnership Act • • • su^^ests that vocational education might 
successfully adapt them to its pperatiohs. • • • Fourth, the existence of 
Federal policy aiid Federal funding for vocational education offers the 
opportunity for crafting a stronger performance orientation [through] 
policies [that] would not require a new Federal initiative and new 
funding. . . . - 

The principal objective of this Background Paper is to examine the possibility of 
instituting more strmgent requirements for outcome-based performance measurement 
than are currently featured in the Perkins Act. While there can be many types of 
outcome measures for vocational education, OTA concentrated on two that are most 
fi^eqiiently mentioned. The first Is based on labor n^arket outcomes, which are intuitively 
appealing because they link important objectives of vocational training to the subsequent 
work experiences of program participants and graduates. Productive and gainful 
employment ranks high among the objectives of American secondary schools, and is the 
traditional raison d'etre for vocational education and training. Job placement, earnings, 
unemployment, productivity, and employer satisfaction are all assumed to be affected, to 
some degree, by the quality of an educational program. 

The second type of measure OTA. examined is based on estimates of learning by 
students. Regardless of their labor market experience after school, hovy much students 
learn, and in which subjects, are important indicators of an educational program's 



8^ E. Gareth Hoachlander, "Performance Based Policies in Vocational Education," 
paper prepared, for the Research Conference of the Association for Public Policy and 
Management, Seattle, WA, October 1988. 
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quality*. In fact,^many people believe that testing of competencies is the single rr.ost 
important.compbnent of performance measurement* 

Job-related competencies and subsequent labor market success are not the only 
accepted objectives of vocational education. There are a number of other measures :hat 
could be included in a comprehensive approach to performance evaluation* For examE-le, 
some research has examined noneconomic effects of vocational education, such as job 
satisfaction and family stability*^ Reduction of the dropout rate and participation in 
postsecondary education are two other outcomes often cited by experts*^^ 

SCOPE AND OBJECTIVES OF THIS BACKGROUND PAPER 

* The House Committee on Education and Labor asked OTA to examine the relative 
merits of alternative performance measures for secondary vocational education* In 
perticuior, the Committee was concerned with two key issues; the technical feasibility 
of such measures, i*e*, questions of data availability, accessibility, and commensur ability; 
and- the appropriateness of various measures, i*e*, the extent to which they could provide 
valid information without distorting the goals of vocational programs or the behavior of 
program participants and personnel* 

The central question addressed in this Background Paper is this: if Congress 
wanted to mandate performance measurement, for the purpose of diagnosing problems in 
specific vocational programs and/or as criteria for Federal funding, what would be the 
strengths and weaknesses of the two most frequently-mentioned strategies? 

Chapter 2 addresses the use of economic indicators to measure secondary 
vocational program performance* OTA explored the available daia that demonstrate the 

9* See, for example, Lawrence Hotchkiss, National Center for Research Into 
Vocational Education, Noneconomic Effects of Vocational Education (Columbus, OH: 
Ohio State University, 1987)* 
10* See Stern, op* cit*, footnote 5* 
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relationship between vocational. education and subsequent labor market performance, and 
then considered the -possible uses of State-collected wage records data as a basis for 
outcome measurement. To gain some preliminary insight into the technical problems 
associated with using the wage records, OTA applied the method in a midwestern school 
district. In addition, this section reports on State efforts at data collection and analysis, 
based on responses to an OTA telephone survey conducted in Dedember 1988. 

Chapter 3 looks at occupational competency testing. OTA's analysis concentrates 
on two key questions: Do tests of occupational aptitude and competency predict future 
labor market performance? Do these- tests provide reliable indications of program 
performance? The role of the States in measuring competency is highlighted in this 
chapter. 

While OTA did not examine formal methods of measuring other plausible outcomes 
of vocational education, these do warrant further attention. Chapter 4 of the report 
outlines.some of the basic issues surroundii^ these measures. 
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Boxl 



Mandating Performance Standards in Federal Programs: 
The Job Trainii^ Partnership Act (JTPA) 



unemployed changed. From 197S-to 1982, the Comprehensive 
Employmeht and Training Act (CETA) was the Federal 
Government's vehicle for "manpower" training. Though CETA 
was a consolidation of numerous pieces of legislation enacted 
in the 1960s, it had no more stability than its predecessors. 
In its brief history, CETA was amended 8 times, had 12 
different titles, and went through 2S separate 
appropriations. The instability of the program's design and 
funding resulted from, the diversity of its objectives: at one 
time or another CETA attempted to remedy the adverse 
effects of automation, retrain experienced workers, create 
jobs, reduce juvenile delinquency, encourage high school 
completion, and conserve natural resources.^ ^ 

With the legacy of CETA's problems, and with the cost- 
cutting consciousness of the "New Federalism" at hand, JTPA 
took shape during the recession of 1981. Championed by 
Senators Edward Kennedy and J. Danforth Quayle, JTPA was 
intended to be a more efficient job training system for the 
poor, designed to operate on less than one-half the funds of 
CETA, with control given to private business and State 
governments. Through a partnership between local 

11. Robert Guttman, "Job Training Partnership Act: New 
Help for the: Unemployed,". Monthly Labor Review, vol. 106, 



During the 1980s, the philosophy of job training for the 



March 1983, p. 3. 



(continued) 
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government > and business^ represented -by private industry 
councils (PIGs)^ JTPA puts the design and administration of 
training programs at the local levels known as service 
delivery areas (SDAs). Where CETA relied on an army of 
auditors to monitp^ compliance with a multitude of method 
and access requirements^ JTPA uses standards based on the 
)abor market outcomes of participants. The Federal role is 
limited . primarily to prescribing effective and enforceable 
performance goals. 

According to the JTPA legislation, the Secretary of 
Labor is to prescribe performance standards to measure the 
increase in employment and earnings and the reduction in 
welfare dependency resulting from participation in the 
program. The Secretary must also prescribe standards 
relating to gross program expenditures.. This is combined 
with a hierarchical management system where all levels of 

* JTPA play a role in determining whom the system serves. 
The Department of Labor has developed adjustment models 
that are intended to hold SDAs "harmless" for serving 
individuals with characteristics that make them hard to serve 
or difficult to place. States play a major role by adding other 
standards, granting incentive awards to SDAs for exceeding 
standards and for serving particular hard-to-serve groups. 

.^Ultimately, it is the SDAs, the PICs, and the service 
providers that respond to these incentives and determine who 

(continued) 
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is actually served in the program. All levels play imp'^rtant 
roles in determining the extent that the hard-to~seive are 

10 

provided training opportunities in JTPA/* 



12. Burt S. Barnow and Jill Constantine, Using Performance 
Management to Encourage Services to Hardrto-Serve 
Individuals in jfPA, Research Report No. 88-04 (Washington, 
DC; National Commission for Employment Policy, April 
1988), p. 48. 
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Chapter 2 

Economic fudicatCi^s of Program Performance 



The impacts of vocational education on earnings^ hours worked, unemployment and 
other labor market outcomes have been studied by economists for many years. ^ Their 
research typically relies on survey data that tracks the educational and employment 
experiencies of large numbers of high school students over several years. These studies 
have attempted to address the general economic question — does vocational education 
pay? — but the data they have relied on can provide very limited information about the 
relative quality of specific vocational programs. Disaggregated, program specific data 
are necessary to ascertain whether students in a given high school program experience 
subsequent labor market advantages as compared to students who are not in that 
program. 



STATE AND LOCAL EFFORTS 



In January 1989, OTA conducted a telephone survey of State directors of vocational 
education to determine the extent of State activity in performance standards. States 
were asked both about placement rate and competency-testing activity, as well as about 
efforts to introduce academic material into the vocational educ&tion curriculum. OTA 



13. Seej; for example, Russell Rumberger and Thomas Daymont, "The Economic Value 
of Academic and Vocational Training Acquired in High School, "Vouth and the Labor 
Market: Analyses of the National Longitudinal Survey, Michael E. Borus (ed.) 
(Kalamazoo, MI; W.E. Upjohn Institute for Employment Research, 1984); Paul Campbell 
et al., Ohio State University, National Center for Research in Vocational Education, **The 
Dynamics of Vocational Education Effects on Labor Market Outcomes," monograph, 
1987; John Bishop, "Academic Skills and Occupational Training," iii U.S. Department of 
Education, Design Papers^ for the National Assessment of Vocational Education 
(Washington, DC?! February 1987); and Robert Meyer and David Wise, •High School 
Preparation and Early Labor Force Experience,?* The Youth Labor Market Problem: Its 
Ndturef Causesy and Consequences, Richard B. Freeman and David A. Wise (eds.) 
(Chicago, IL: University of Chicago Press, 1982). 
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found that States use placement rate data within a Urger framework of accountability, 
which usually includes information about a number of variables related to vocational 
education programs. Examples of these variables, many of which are specified in the 
Perkins Act, include access, and support services Tor handicapped and disadvantaged 
students, appropriateness of equipment and supplies, safety of work areas, 
appropriateness of curriculum, existence and operations of advisory committees, 
provision of guidance and placement activities, and staff development. 

Many States and school districts attempt to evaluate their vocational programs by 
tabulating labor market statistics of graduates with information from their transcripts. 
As shown in figure 2-1, 27 States use followup surveys to collect data on job placements 
of graduates, which in many oases includes information on wages, job tasks, and 
employers' views of employee job performance. Followup surveys are typiceily 
conducted 9 to 12 months after the student has left school and data is generally collected 
by district personnel and/or teachers. 

Only nine States have established policies thpA directly tie placement rates to the 
distribution of funds. The established rates range from 50 to 70 percent, with three 
States requiring 50 percent placement four requiring 60 percent, one requiring 65 
percent, and one requiring 70 percent. Seventeen States use placement-rate data for 
program evaluation but do not link the rates to fundir^ decisions. Six States are planning 
to use piacement-rate data, of which one is considering required rates. The States that 
mandate placement rates usually provide technical assistance to deficient programs 
before withholding funds. 

To gain additional insight into the implementation of placement rate standards 
OTA looked more closely at the policies.of five States. 



Figure 2-1 
State Use of Pfacement Rates for 
Vocational Program Evaluation 




I 



States collecting placement-rate data 
for program evaluation only (includes DC) 

States considering or planning collection of 
placement-rate data 



□ States not currently collecting 
placement-rate data 



> w 



states with required placement rates 
tied to distribution of funds 



State required placement rates: 

50% - Alabama, Kansas, South Carolina 

60% - Florida, Missouri, Ohio, West Virginia 

65% - Kentucky 

70% - Massachusetts 




SOURCt; OTA telephone survey, January I9tt9. 
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Missouri 

Since 1985^ approximately 37 percent of Missouri's State funds for vocational 
education have been distributed to programs according to scores on two factors: 1) how 
well programs train students for jobs in arees of high market demand (demand is 
estimated from rankings prepared by the Missouri Occupational Information Coordinating 
Committee in cooperation with the division of Occupational and Adult Education of the 
Elementary and Secondary Education Department, using employment projections 
prepared by the State Employment Security Agency), and 2) placement of students in 
jobs related to training. Local placement performance is accepted as sufficient 
justification to continue a program that is identified as being of low priority using this 
ranking procedure. A bonus is offered for placing hard-to-^erve students. The State 
provides districts with information regarding the dollar value of each of their programs 
based on the above two factors, thereby fCiving districts information needed to make 
maintenance or cancellation decisions. 

in keeping with Perkins Act requirements, programs are also evaluated once every 
5 years on process factors, such as whether they have their competency-bused 
educational program in place and whether they have an advisory committee. These on- 
site evaluations, however, do not influence the distribution of vocational education 
fundJ. Missouri has not yet conducted a formal evaluation of the impact of their 
placement-rate policy. 

Kansas 

Kansas requires that 50 percent of its vocational education graduates find jobs 
related to the field in which they were trained. Students who have dropped out in the 
11th or 12th grades can be counted as placements if the jobs are in the area of training* 
Programs that do not meet the 50 percent requirement are placed on probation and have 
3/ years to raise their rate. During those 3 years, the program receives technical 
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assistance from the State Education Departmenti and if a 30 percent placement rate is 
still not achievedi support is discontinued. Less than 5 percent of the programs in Kansas 
have failed to meet the standards. 

South Carolina 

South Carolina uses both positive and negative incentives to influence vocational 
ediiesation. The 50 percent placement requirement means that one-half of all students 
who complete defined courses are expected to find jobs. This requirement is different 
from those in effect elsewhere that are based on progra^i, school, or district 
completions. Placements must be in the area of training and attendance at a 
postaecondary institution must also be in the area of training. South Carolina also 
includes dropouts in Its count of successful placements if they obtain jobs in the area of 
training. Placement rates are verified by the State^ and involve pulling of records, visits 
to employers, and interviews with employers and graduates regarding wages and 
employee satisfaction. 

As a positive incentive, South Carolina's General Assembly allocated in 1984 $2b 
million to vocational education programs to be distributed over a S-ye&r period for the 
purchase of high-technology equipment. Districts are required to do a needs assessment 
every 5 years, based in part on labor market conditions, which is to be used in their 
proposal for funds to purchase the high-technology equipment. District plans and their 
needs assessment reports are read by a private sector panel. South Carolina also places 
strong emphasis on the acac emic aspects of vocational education and participates in a 
vocational education eonsortiun. of the Southern Regional Education Board, whose 
pri.nary aim is to promote the infusion of academic material into vonational educat:on.^^ 



14. See ch. 4 for a more detailed description of the* Southern Regional Education 
Board's performance measurement strategy. 
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- An average^of seven programs nave been discontinued in each of South Carolina's 
92 districts over the past 4 years. Discontinuances are reported to be based primarily on 
decline in labor market demand rather than on inadequate program quality* For example, 
♦programs in body and fender and building construction have been discontinued. The 
discontinuances are based on a combination of low placement rates and failure to qualify 
for the high-technology equipment funds. 



Florida*^ legislature enacted a placement rate requirement of 60 percent in 1983- 

1984, then raised it to 70 percent in the next year. Prc^rams that fail to meet the 

standard are automatically reviewed by the Florida State Department of EuuOatlon. 

which then submits the review to, a regional coordinating council along with a plan for 

• 

improving the placement rate for that. program. As of the 1985-1986 school year, any 
program in which the placement rate for completers was less than 70 percent for 3 
consecutive years was ineligible for State furding. To compensate for adverse economic 
or demographic conditions, the Department of Education was given the authority to 
adjust placement rates at the regional level using a statistical method approved by the 
State board. Adjustments must be reviewed by the Legislature before submission to the 
State board. 

Florida's unemployment rate for youth in 1987 was 16.1 percent. The placement 
rate requirement of 70 percent estauiishes the expectation that completion of a 
vocational proe'^am should increase the likelihood of job placement by approximately 14 
percent- over what would occur without special training. 

Data on placements is collected at the local level, generally by the vocational 
"program teacher .with the assistance of a placement specialist. This data is then 
forwarded to the district and on to the State. The Florida Legislature specified that 
audits of accuracy regarding placement rates are to be conducted by the Office of the 



Florida 
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^Auctitor-General of Flprida^^ although none have been conducted.as yet* Florida's response 
rate on these ;&)llowup surveys has increased from 62 percent in 1982-1983 to 89 percent 
inl985-1986. f 

Florida is moving to a statewide system for tracking individuals who complete 
vocational programs. This system draws from employment data, postsecondary 
enrollment, and national military records, and uses .students' social security numbers as 
identifiers. One limitation is that not all students have social security numbers and 
cannot be required to obtain or report one. Once this data is collected, it must be 
returned to the local ediication agencies, who then determine whether or not the 
placements are in areas related to training. 

Florida has defined a voe^tional completer as a. student who has completed a 
specif ied course of study and exhibits mastery of designated competencies. Exceptional 
students are not included in the computation of program placement rates if they do r^t 
meet these criteria. Also excluded from the computations are prison inmates, 
nonresident aliens, or aliens who are in the United States on nonimmigration visas. 
Students who are not located are counted as nonplacements. 

In 1983-1984, 45 percent of the total prc^ams were required to receive a modified 
program review, because they had not met the place ment*rate standard. Th^s rate 
dropped to 14 percent during the next 2 years. The number of students (both secondary 
and postsecondary) who were designated as having completed vocational programs 
declined by almost 12,000 during this same period. Possible reasons are that schools 
either are being more cautious about who qualifies as a completer, more selective 
regarding who is permitted to enroll in a vocational program, or that fewer students are 
electing to errull in vocational programs. In addition, these same data show that, while 
placements in employment related to training increased, placements in continuing 
education and full-time military dropped substantially. No studies have been conducted 
to^identify possible explanations of these trends. 
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fllinois 

Sevfin "vital signs" comprise Qllnois' model of vocational program evaluation. The 
vital signs, based on a medical model of diagnosis, include labor market justification, 
placement, enrollment, employer satisfaction, student satisfaction, student performance, 
and cost containment. A particularly interesting aspect of niinu^c' accountability effort 
is their i^eliance on a computerized expert system that helps to identify causes of 
problems. Plans for improvement are designed collaboratively with local school 
personnel and State education staff, and are derived from the information gathered and 
analyzed by the./^xpert system. If this system is demonstrated to be accurate, it could 
become attractive in other States, where the basic computerized model could be tailored 
to meet local needs. 



WAGE RECORDS DATA AS A BASIS FOR PERFORBfANCE MEASUREMENT 



State and local efforts at performance measurement, while commendable, 
frequently fall short of providing accurate and statistically valid indicators of program 
quality. First, the data are self-reported by program graduates, who m^y ha je difficulty 
recalling the courses they took in school and who may^bo unwilling to give accurate 
earnings information.^^ Second, the surveys are usually conducted by school personnel. 



15. For example, in a study that matched transcript data against self-reported 
information, it was found ttiat many students misclassified themselves: "... a number of 
those who reported themselves to be in the general track in high school actually took 
three or four vocational courses." See John Bishop, "Policy Evaluation With Archived 
W&ge Record Data: Limitations of Existing Data Sats," in Northeast-Midwest Institute, 
The Feasibility of a NatiomL Wage Record Database: Four WorJdng Paper, prepared for 
a conference on "Employment and Earnings Dynamics in the United States: Policy Issues 
and a Longitudinal Data Source" (Washington, DC: January 1989), p. 14. Also see Robert 
H. Meyer, "An Economic Analysis of liigh School Vocational Education, I: Vocational 
Educatiom How Should It Be Measured?" (Washington, DC: Urban Institute unpublished 
iuouOgraph> Aug. 31, 1981. 




which introduces a possible bias if program funding is tied to survey responses. Finally, 
most ol.theseisurveys have .relatively low response rates, because graduates are hard to 
locate.^^ 

These kinds of technical and methbdolo(;ical problems have always made vocational 

educators wary of performance measures based on economic outcomes. But recent 

changes in the way States gather eamir^s data have spurred renewed interest in the 

possibility of relatively cost-effective outcome-based performance measurement. As a 

result of amendments to the Social Security Act contained in the 1984 Deficit Reduction 

Act (Public Law 98-369)9 States now maintain p** * some and eligibility verification 

17 

system based on employers' quarterly wage reports. For the first time, fourth quarter 

earnings for 1988 will be available for all 50 States, from which it will be possible, in 

18 

principle, to construct longitudinal earnings profiles for^ almost all working adults. 
Covered employers report wages for employees who were on the payroll during the 
quarter, inclu£hg the value of meals, lodging, and other remuneration*. Thus, for 
example, eanungs data for a June graduate, whose first full qciarter of employment 
begins in July^ would be avaiJable for retrieval beginning in January of the next year. 

While there are possible sources of- error in these data, due to reporting 
inaccuracies of various sorts, the v^age records are t /oically quite reliable, and are 
considered greatly superior to self-reported information. !?ecause they are employer- 
based, these data have the added benefit of linking earnings information with 
characteristics of reporting companies. In addition, because States pay unemployment 
insurance based on these data, there are strong incentives for accurate and timely 



16. In New York City, for example, the only graduates surveyed are those whose 
telephotie numbers are the' same as they were in school. 

17v This discussion-, is based oii David Stevens,. "Using Wage Records Data to Construct 
Measures of Secondary Vocational Education Performance," CTA contractor report, Dec. 
12, 1988. 

18. There are exclusions- that do not severely hinder the usefulness of the data for 
secondary^ vocational -education -program assessment. See Stevens (ibid.) for a more 
detailed? discussion of wage records coverage. 
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reporting* (Most States .determine eligibility to receive unemployment compensation 
benefits based on covered earnings in the first four of the last five completed quarters.) 
While there is some justified concern with confidentiality issues, most knowledgeable 
observers conclude that appropriate usv^s of the data can be designed following strict 
.guidelines tcprevent dsclosure of personal wage and employment information^. 
The principal advantages of the wage records data are: 

• coverage is comprehensive enough for most trackii^ purposes; 

• the data are available quickly enough for most types of evaluation; 

• the data are more accurate than self-reported survey information; 

• the data can be merged with other databases (such as military service 
records or education records), using social security numbers for 
student identification and then deleting individual identifiers for 
analysis purposes; and 

• movement across State boundaries can be traced by establishing 

procedures for matching interstate records (several employment 

security agencies have already done this for their own administrative 
19 

purposes. 

Despite these improvements in the quality and cost of data collection, however, the 
usefulness of the wage records system is limited by problems that exist with any kind of 

19. See James Hanna, "The. State Perspective on a National Wage Record Database," in 
Kortheast-Midwest institute, op* cit*, footnote 15* 
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labor> market data* basic- question js whether economic" outcomes provide enough 
information to measure fully* program performance. Many vocational educators are 
opposed to relying on these measures because of the difficulty in isolating the effects of 
.participation in a vocational program from other factors that influence labor market 
performance.^^ For example, if individuals who choose to enroll in vocational programs 
are more interested and/or more experienced in specific skills than those who do not 
enroll, then it is difficult to measure the added value of participation in the-program. 
Similarly, use of labor market outcomes requires, in addition to substantial background 
information on program participants, detailed information about employer recruitment 
and compensation practices, which may not always reflect accurately on either individual 
abiliti^ or program quality. In addition, there is concern that emphasis cn placements 
will induce schools to concentrate resources on students who are most likely to be 
placed, at the expense of those who need the most instructional attention. (The problem 
of "creaming/* i.e., program managers* incentives to V^ain the most likely-to-succeed 
individuals, is discussed in box 2, page 36). Thus, while the wage records offer the 
potential to assess relatively long-run labor market effects of vocational programs, there 
will likely always be a credibility problem. 

There remain also some technical problems that need to l;^ resolved to allow 
effective use of the wage records data (see also box 3, page 42): 

♦ Administrative information about high school programs is not 
uniform. Individual institutions and school districts use unique 
classifications of vocational and academic courses, as well as different 
definitions of vocational students. While externally imposed 

20. A recent national poll found that nearly 70 percent of the 265 responding school 
districts were opposed to the use of job placement rates as a measure of student 
success. See National School Boards Association, Office of Federal Relations, "Survey on 
Vocational Education; Summary and Recommendations for Reauthorization of Carl D. 
Perkins Vocational Education Ac\," unpublished document, February 1989. 



" uniformity* could solve the comparability problem^ it might also drive 
out important differences in curriculum and placements that stem 
from local school authorities' knowledge of local conditions. 



Although there, are no legal barriers to the use of earnings data — 
provided that individuals are not identifiable — the confidentiality 
question continues to pose problems. Some administrators, for 
example, who are uncertain about legal restrictions, take a cautious 
route and prefer not to make information available from earnings 
files. Relieving these uncertainties would be an important prerequisite 
toward-wider utilization of the wage records data. 

The efficiency of the wage records approach is partly a function of the 
administrative costs associated with this kind of data management. 
Currently, the merging of data on students^ exposure to vocational 
courses, their acquisition of various skills, and earnings and 
employment data requires coordination of at least two State agencies 
(Employment Security, which manages the wage records data, and 
Vocational Education, which collects school data). 
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Box 2 

JTPA Revisited: Some Effects of Outcome Measures 



The Job Training Partnership Act and its performance 
standards system have been operating for nearly 8 years and 
are considered: by many observers to be remarkably 
successful. Labor Department officials note that almost 70 

91 

percent of trainees have found jobs. Other supporters, m 
both political parties, claim that JTPA is proving to be a 
cost-effective tool for tralnii^ and job placement. But a 
group of critics, equally bipartisan, argue that while the 
program is a significant improvement over CETA, its 
emphasis on cost and placement has shortchanged the very 

on 

poor and hard-to-employ. - The performance standitirds 
create financial incentives for program managers to place 
applicants quickly in private sector jobs and at low cost. 
"[JTPA] pretty much serves the job-ready individual as 
opposed to the long-term unemployed, does not adequately 
reach the problem of youth unemployment and is severely 
limited in the number of people who are eligible to take 
advantage of training," said Rep. Augustus F. Hawkins, 
chairman of the House Committee on Education and Labor. 



21. Neal R. Peirce and- Robert Guskind, "Job Training for 
the- Hard-Core Unemployed Continues to Elude the 
Government," National Review, vol. 17, Sept. 28, 1985, p. 
2197. 
22i Ibid. 

23. Ibi' . . 2198. (continued) 
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•The issue of who is eligible for JTPA services brii^s up 
the problem known as creaming, which describes a situation 
where a service delivery area (SDA) might tend to enroll 
clients who are easiest to train and place because it can be 
done most costreffectiyely. John Ev Eisher, general chairman 
and chief executive officer of Netionwide Insurance Company 
and chairman of the Columbus-Franklin County (Ohio) private 
industry council (PIC), which provides basic skills and 
remedial programs for the least-qualified applicants, says 
"the performance guidelines are unintentionally pushing the 
program towards creaming, to be sure."^* But others, such as 
Senator James M. Jeffords, (R-Vt), counter that with limited 
funds it is bound to happen, "Human-^nature tells you to serve 
those who come to you rather than people who have to be 
kicked in the rear."^^ This sentiment is echoed in another 
PIC chairman's job training strategy: "Find real, live jobs 
that clients can fill without going back to school. If they're 
illiterate, there are plenty of funds available for remedial 
education. If they've dropped out of high school, they'd 
better go back." 

In a recent report on the effects of JTPA performance 
standards on clients, services, and costs, the National 
Commission for Employment Policy (NCEP) found that in 
areas with loy^ unemployment, most of the clients interested 

24. Ibid. 

25. Ibid., p. 2199. 

26. Craig Mellow, "Motown's Manpower Renewal," Across 

the Board, vol. 24, June 1987, pp. 31, 34-39, p. 36. (continued) 
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in JTPA-programs w.ere thoserwho. had little work experience 






. or had major barriers to employment^ often requiring basic 






skills remediation. In contrast^ in areas with high 






unemployment, those seeking JTPA services generally had 


fx' ■ 
J'* > 




more job^ skills and required help in -retraining for new 

9.1 

industries or occupations. 






The NCEP report found that while performance 






standards can influence the type of clients enrolled in JTPA, 






the effectis are not large. Even in States with policies that 






are found to discourage services to welfare recipientSi SDAs 


f ,1* 




are enrolling a considerable number. For example, on 
average, 29.5 percent of adult clients in JTPA programs are 


< , 




welfare recipients, approximately equal to the^r incidence in 
the eligible population, which is 30.6 percent. The results of 
the evaluation indic&te that even in States with Dolicies that 


t 




discourage service to hard-to-serve clients, SDAs are serving 






these clients only slightly less than their incidence in the 


/-^^ ^ 
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population. 


Si' 

t 




Many researchers agree that in a human resource 
development program where the mission is clearly defined, a 
well-targeted set of performance standards can have the 
desired effect of making the process as efficient as possible. 


i • 




While well-defineU goals can make a performance 




management system more likeiy to succeed, there is also 


fit . ' 




'27. National Commission for Employment Policy, 
Evaluation of the Effficts of JTPA Performance Standards on • 
ClientSf ServiceSf . and CostSf Research Report No. 88-16 
(Washington, .DC;: September 1988), p* 119. 

28. Ibid., p. 73. (continued)' ' 
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— evidence in. JTPA of an. emerging conflict between the goals 
of -enrolling the hard-to-serve and encouraging effective 
programs. ^ "By definition SDAs take risks when they enroll 
hard-to-serve individuals, and if the performance standards 
system does not include adequate adjustments in the level of 
expected performance for serving these individuals, SDAs 
face a tradeoff between enrolling the hard-to-serve and 
achieving a high level of measured performance."^^ 

In an attempt to manage this tradeoff, there has been 
increasing interest in defining hard-to-serve status in terms 
of deficiencies and barriers, such as lack of basic skills and 
minimal work experience, and incorporating these factors 
into the adjustment models. From this, standards can be 
developed to encourage services to* bard-to~serve 
individuals. The inclusion of a new youth standard in 1988, 
called "employability enhancement," attempts to measure 
such things as work attitudes and job search skills. This 
expansion in performance standards recognizes that desired 
outcomes cannot always be measured simply by a placement 
rate. 

The relative success of JTPA may lie in its clear and 
well-defined goals and objectives. JTPA's mission is to move 
individuals and families toward economic self-sufficiency. 
Unlike vocational education, which provides both technical 
and nontechnical training to "students" who plan to be both in 

29. Ibid. Barnow and -Cons tan tine, op. cit., footnote 12, 



p. 3. 
30. Ibid. 



(-continued) 
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^•^ • and out of the labor* markiti job training has re3tricted itself 
'.to more easily defined goals; When a consensus about program 
mission and goals exists, as in JTPAi appropriate measures of 

31 

performance can be more easily designed. 

Differences in leverage and control through funding also 
set vocational education and JTPA s^art. With nearly all 
JTPA's appropriations coming from the Federal Governmenti 
the use of incentives and sanctions da^^ed on the performance 
measures is a serious motivational i^^U Secondary vocational 
education^ on the other hand, has only a small percentage of 
its funding from outside State and local sources, substantially 
limiting the Federal Government's financial leverage. 

There may be sohij important lessons for vocational 
* ' education from the Federal, State, and local experiences with 
JTPA. In additio'" to the basic need to clarify program 
objectives, it is important to keep in mind that: 

« performance standards appear to work well 
if local authorities are able to meet their 
other performance goals without major 
revisions; 



31. Christopher T. King, Cros^Cutting Performance 
Management Issues in Human Resource Programs^ Research 
Report No. 88t12 (Washington, DC: -National Commission for 
Employment Policy, August 1988), p. iv. (continued^ 
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♦ • -incentive policies-that- reward programs for 

going beyond defined outcome standards 
may lead to reduced service for some hard- 
to-serve groups; and 

• Federal standards based on minimizing 
program costs could have the worst impact 
on provision of services to hard-to-serve 
populations.^^ 



32, See National Commission for Employment Policy, 
Evaluation of the Effect^' of JTBA Performance Standarda on 
Clients^ Semces, - and - Costa; - Ex2cutive Summary 
(Washington, DC: September 1988), pp. 4-5. 




Box 3 



. Merging Wage Records and Transcript Data: 
A ^monstration 

In November 1988} OTA conducted a small study to 
demonstrate the potential of merging wage records data and 
vocational education student transcripts. OTA had access 
to all third quarter wage records for the years 1982 to 1987, 
from a single State Employment Security Agency* Tk^se data 
represent an example of what couiu be available at the 
national level if a national wage record archive or a 
distributed network ot State Agency administrative records is 
established. 

To complement the 6 years of wage data, OTA obtained 
social security numbers of 138 secondary vocational 
graduates from the class of 1984 of a local vocational high 
school. Assurances of strict confidetvtiality were required, 
including no use of names and the deletion of 'Jl social 
security numbers from the final analysis file. 

The social security numbers were matched against each 
year's wage record file. The table below shows tiie "hit" rates 
that were achieved in matching the transcript data and the 
wage records, along with an ea.^ings profile for the third 
quarter of each year. 



33. The study was conducted by DavM Stevens of the 
University of Missouri-Columbia, under contract with OTA. 
See David Stevens, "Using Wage Records Data to Construct 
Measures of Secondary- Vocational- Education Performance," 
OTA contractor report, Dec. 12, 1988. _ 
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C jUARTERLY EARNINGS ** 

Percent of 



Tear 


"hits"* 


Mean 


Minimum 


Maximum 


1982 


12 


$ 718.32 


$40.20 


$1,512.95 


1983 


63 


965.92 


14.88 


3,991.58 


1984 Graduation 


74 


1,585.01 


37.51 


5,451.73 


1985 


63 


2,284.65 


27.99 


6,164.00 


1986 


64 


2,455.2?. 


24.38 


7,225.17 


1987 


67 


3,035.94 


61.42 


7,077.83 



, *Employed anywhere in the State during the third quarter of the designated year. 

Total earnings reported by all covered employers for whom an individual worked 
during the relevant quarter. 



This table demonstrates the ease with which 
longitudinal earnings data can be retrieved for those who 
complete secondary vocational education programs. 
Quarterly data can be summed to provide annual reported 
earnings figures. Any classification system of school courses 
or achievement indicators could be merged with the fi^^s 
making it possible to evaluate vocational programs at the 
district, school, or even class level. External activities, such 
as military service, Federal Govrrnment employment, or 
enrollment in postsecondary education, could also be added. 

Geograpiiie information in the wage records file permits 
the location of the graduates' places of employment. Where 
major metropolitan areas cross State borders, hit rates 
relying on a single State's data will be reduced accordingly; 
interstate matching of wage records can resolve this 
problem. If a national wage records archive existed, it would 
be ..possible to match social security numbers against other 
States' records to detect reported employment in any other 

(continued) 




state. Two other types of information, the propensity for 
employees, to remain with a given employer and/or in a given 
industry, can be derived using any ting and ending 
dates/'* These data are^important if job retention is selected 
as an outcome measure; 

OTA did encounter several problems stemming from the 
ad hoc nature of the study. For example, the initial request 
for program completer transcripts also included an academic 
high school in the area, but social security numbers had not 
been maintained or the permwient student records. This 
problem would not recur if maintenance of social security 
numbers or other identifiers was mandated. (Note that due to 
recent changes in the tax law almost all children are 
acquiring social security numbers.) Many schools already 
have automated record systems? and most can be expected to 
adopt them in the next few years. Errors in data 
transcription at the school will increasingly be eliminated by 
the elentronic transfer of information. It was also not 
possible to identify simultaneous enrollment in postsecondary 



34. In other applications, Stevens has prepared matrices of 
intercounty and interindustry movement that reveal patterns 
of "staying," "exit," and "entry" affecting the specific sectors 
that are of interest. Tracing these flows is important in 
attempting to understand the role of secondary vocational 
education in economic development dynamics: who stays and 
who . leaves, and how . does each prosper through 
employment? See Stevens, op. cit., footnote 17. (continued)'* 
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education, and- reported employment. This would require a 
separate matching -of social security numbers against each 
higher education institution's records. 

OTA also discovered confidentiality issues in this case 
study that could pose problems if the wage records system 
were implemented for performance measurement. For 
example, in 14 cases the data merge yielded only 1 graduate 
employed in a given county. This would reveal the identity of 
the individual to anyone who knew that a graduate of that 
class was working i:i that county. While that may seem to be 
innocuous information, the same observer might also cross- 
match this geographic data with industry-specific 
employment data containir^ earnings figures. Industry- 
specific "stayer" rates, i.e., the percentage of workers who 
remain in a given industry ovar time, present a frustrating 
confidentiality problem as well. While it would be useful to 
know how a new enterprise is staffed or the destination of 
workers affected by a plant closing or layoff^ this information 
could be considered private. These questions would need to 
be addressed without revealing individual or other employing 
unit identities. There are ample precedents for devising 
procedures to comply with existing privacy laws (e.g., the 
Bureau of Labor Statistics, the Social Security 
Administration, and the Internal Revenue Service). 
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Chapter 3 

Heasurement of Occi^ational Competencies 

What students learn in school is at least as important an indicator of program 
quality as how well they do !n the labor market after graduation. OTA examined the 
suitability of using various types of occupational competency tests for vocational 
program assessment. One key assumption implicit in the use of these tests is that the 
knowledge arid skills they measure are correlated with one or more of the objectives of 
vocational education . Although vocational education has many objectives, the 
preparation of young paople for productive and gainful employment ranks high. Thus, if 
mechanical ability, for example. Is known to have a positive impact on the average 
earnings of workers in certain 0( eupations, then performance on a test of mechanical 
competency can provide a measure of the quality of the educational program in which 
that competency was taught. (Trainii^ students in obst^lete skills is not the sien of a 
quality program, no matter how well the skills arc taught.) OTA examined the 
relationship between performance on a test of occupational competency and subsequent 
labor market outcomes. 

Linking certain developed abilities to the probability of attaining variolas objectives 
of schooling, however, is insufficient for performance evaluation purposes. The second 
key assuTiption in the use of competency tests is that they provide information about the 
effects of school programs . Ideally, a test's items woultj be keyed exactly to the content 
of specific courses, and performance on the test would be dn accurate gauge of how well 
those courses were taught. In practice, however, no test can fully sort out the effects of 
prior knowledge, prior experience, arid innate ability from participation in a specific 
course or program.** And just as placements and earnings are insufficient indicators of 

35. This argument is developed in greater detail in Alexandra K. Wigdor and Wendell R. 
Garner (eds.). Ability Testing: U[ses, Consequences^ and ControversieSf Part I: Report of 
the Committee (Washington, DC: National Academy Press, 1382). See esp. pp. 25-29. 
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program quality because they confound program effects and individual or environmental 
attributes^ scores on. even the most detailed^ content-specific tests provide only partial 
measures of program quality. OTA explored the relative merits of various types of tests 
— many of which are already available and in use in different parts of the country — for 
performance measurement. 

APTITUDES AND COMPETENCIES 

While aptitude tests are designed principally for assigning individuals to jobs and 
training programs for which they are well suited^ their questions are often quite similar 
to those that appear on tests of achievement. As the National Academy of Sciences 
explained in a 1982 report: ". . . aptitude tests^ intended to predict what a person can 
accomplish with trainings and achievement tests^ intended to measure accomplished skills 
... are not fundamentally different. They both measure developed ability, they ofter 
use similar questions, and they have often been found to yield highly related results."^^ 

OTA explored the possibility of applying a widely-used test of occupational 

on 

aptitude as part of a comprehensive system of program evaluation. The Armed 
Services Vocational Aptitude Battery (ASVAB) is a 3-hour test administered by all 
branches of the military to dete nine eligibility of recruits and to match rec. \^ts to 



36. Ibid*, p. 27. The National Academy explains how aptitude tests can be used ^ 
measure achievement: *'A test for mechanical aptitude would be included in a battery . 
tosts for selecting among applicants for pilot training since knowledge of mechanical 
principles has been found to be related to success in flying. A similar test would be given 
at the end of a course in mechanics as an achievement test int( to measure what 
was learned in the course. Of course, it would not be surprising tc * i that m any people 
who did well on one of ^he tests would also do well on the >ther, nor that the 
achievement test could also be used to predict flying success." Nevertheless, experts in 
the field of testing and measurement caution that, on the continuum between ability and 
achievement, tests designed to measure the latter are typically better suited for 
performance evaluation than those intended primarily for measurement of at^titude or 
innate ability. 

37.. This- section, draws • heavily on John Bishop, "Occupational Competency as a 
Predictor of Labor Market Performance," OTA contractor report, Dec. 12, 1988. 
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jobs. The orie:inal purpose of. the ASVAB was to predict the success of new recruits in 
technical training; and since 1980 a major research effort has been under way to explore 
the uses of the ASVAB in predicting job performance. 

The test battery consists of 10 subtests in mechanical comprehension, auto and 
shop information, electronics information, clerical checking (coding speed)^ numerical 
operations (a timed test of simple arithmetic), arithmetic reasoning, mathematics 
imowledge (covering the liigh school curriculum), general science, word knowledge, and 
paragraph comprehension. Unlike other competency tests that focus on specific 
occupations, ASVAB technical subtests assess broad technical ability and trainability for 
jobs involving the operation, maintenance, and repair of complicated machinery. The 
ASVAB^does not measure specific proficiencies within these general technical clusters. 
The mechanical comprehension, auto and shop knowledge, and electronics subtests 
correspond roi-ghly to tednical trades, and performance on these tests can be 
interpreted as n indication of competency in the technical arena. Because the ASVAB is 
currently administered annually to 1 million high school students in 14,000 schools, as a 
tool in career counseling, there is interest in the possible uses of the ASVAB in 
performance evaluation. 

Earnings and Employment 

OTA first sought to determine whether vocational subtests of the ASVAB and by 
extension, other similar paper-and-pencil tests of occupational aptitude — are valid as 
early indicators of labor market success in civilian occupations* The analysis is based on 
data from the National Longitudinal Survey. ''^ 

38. During the summer of 1980 the Armed Service Vocational Aptitude Battery was 
administered to 94 percent of ne youth sample of the National Longitudinal Survey 
(approximately 11,000 respono. i). The earnings, wages, and unemployment rates for 
this sample were collected in foUowup surveys conducted in each year through 1986. 
Multiple regression models were used, to control for weeks employed, school attendance, 
years of- schooling, -highest level of schooling^cok "dieted, years of college education, 
minority group status, census region, and unemployn.ent in the local labor market — all 
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. As: shown -ir. figure 3rl, ASVAB subtests :measurmg electronics information and 
mechanical, aiito, and shop knowledge, as weU as coding speed and numerical operations, 
are. positively correlated with, wage rates and earnings, aiid negatively correlated with 
unemployment, niong young men. For men in the National Longitudinal Survey sample, 
there is a very substantial economic return to technical knowledge as measured by these 
subtests, largely because men with vocational education are likely to take jobs in the 
mechanical, blue collar, and manufacturing sectors in which employers value tec^micci 
knowledge. ^ 



Job Performance 

Although economists often assume that earnings are directly correlated with job 
performance and productivity? there is also evidence of divergence between wage rates 
and performance.*'' It is therefore important tc ascertain whether ASVAB scores 
correlate as well with measures of productivity as they do with etarnings and 
employment. 

To address this question OTA used a dataset in which ASVAB subtest scores were 
related to a hands-on meas^ire of job performance.*^ The findings from this analysis 



factors that could influence earnings and employment — as well as academic ability, as 
measured by subtests on arithmetic reasoning, mathematics, science, word knowledge, 
and paragraph comprehension. See ibid., for detailed model specifications. Also see 
Mark J. Eitelberg et aL, Screening far Service: Aptitude and Education Criteria for 
Military Entry (Washington, DC: U.S. Department of Defense, Manpower, Installations, 
and Logistics^ September 1984). 

39. Except for coding speed, the Armeri Services Vocational Aptitude Battery (ASVAB) 
vocational subtests did not correlate with women's labor market performance. These 
differences can be explained by the fact that women who do not attend college are still 
employed mostly in retail sales and clerical jobs where the technical skills included in 
ASVAB are considered less relevant by employers. See Bishop, op. cit., footnote 37. 

40. One study found that ah individual who is 20 percent more productive than average 
earned only 1.6 percent more in initial wages; after 1 year of employment, the more 
productive employees received only 4 percent more in nonunion firms (with about 20 
employees), and experienced no vyage advantage at £ill in unionized establishments (with 
more than 100 employees) or in large nonunionized firms (more than 400 employees). See 
John Bishop, "The .Recognition and. Reward of Employee Performance^" Journal of Labor 
EconomicSf vol. 5, No.^4, part i2, October 1987, pp. S36-S56. 

41. ' See Bishop, op. cit., footnote 37; and Milton Maier and Francis Grafton, U.S. Army 
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Figure 3-1 

Correlation Between Occupational Competencies and Earnings 
of Young Men, 1984-85 



14 



12 - 



o 10 - 



c 

E 
o 



8 - 



4 - 



2 - 





Mechanical Electronics Coding speed Numerical 
Information operations 

(Timed) 

Armed Services Vocational Aptitude Battery (ASVAB) Subtest 
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^ Mechanical Comprehension and Auto and Shop Information 
subtests equally welghtied 



SOURCE: Oiik% of Technology Ass«ssmenf, bos^d on NoHonoi LongMudlnoi Study data 
rtporitd In John Bishop, "Occupoflonal Comp«f«ncy as a Pr«dlctor of Labor Market 
Performance," OTA controctor report, 1988, 
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further support the.validity of the ASVAB as a.predictor of labor market pe/formarice. A 
small increase in all four of the nonclerieal vocational subtests Is correlated with an 
increase in job performance in skilled technical jobs^ skilled electronics jobs, general 
maintenance jobs, mechanical maintenance jobs, missile battery operations, food 
services, combat occupations, and field artillery. Because many of these jobs have 
civilian counterparts, the analysis suggests that technical competencies measured by the 
ASVAB are valid determinants of worker productivity in many civilian jobs. 

Program Evaluation: Caveats 

OTA's analyses show that the AS/AB, and by extension similar paper-and-pencil 
tests of technical aptitude, can provide early indications of future labor market 
performance. For the purposes of this Background Paper, however, a central question is 
whether the ASVAB can provide sufficient information about how measured vocational 
abilities are acquir sd. For if such tests are to be useful in program evaluation, it must 
be possible to trace test performance to participation in specific school programs. OTA 
finds that the ASVAB does not meet this criterion, and that it could not become the sole 
or primary instrument of performance measurement, for the following reasons: 

ASVAB vocational subtests (in auto and shop information, mechanical 
comprehension, and electronics information) consist of a total o^ 70 
questions, to be answered in 39 minutes. They do not provide 
sufficient information to judge an individual's technycal proficiency in 

AO 

the tested skill areas.^^ In addition, the ASVAB does not cover many 



Research Institute for the Behavioral and Social Sciences, "Aptitude Composites for the 
ASVAB 8, 9, 10," Research Report 1308, unpublished manuscript. May 1981. 
42. An analogy with academic t^^ting is illustrative. A general science test that 
consists of 40 questions, 10 each on physics, chemistry, geology, and biology could 
provide a measure of -general scientific knowledge without indicating anything about 
knowledge in any of the specific subjects. 
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--skills that are A taught: Jn American secondary school programs. For 
example, there would be^no way to measure an individual's competency 
; in cosmetology, a field that attracts many women students. 

* -.Participation in high school vocational education programs is only 
weakly ^oorrelated with the ASVAB scores.^^ Given the brevity of the 
vocational subtests this result is not terribly surprising. It suggests 
that using ASVAB scores to evaluate programs would require extensive 
additional data to control for student background, socioeconomic 
status, and other correlates of test performance. These data are 
costly to obtain, and the requisite statistical techniques are highly 
sophisticated. 

For measurement of program performance, then, the ASVAB is an inadequate 
instrument. However, because it does predict future labor market performance quite 
powerfully for some stu^'^rls, it can be useful as part of a broad package of evaluation 
and program improvement. In particular, OTA considers the follov;ing potential uses of 
the ASVAB worthy of additional research: 

The ASVAB can be used to diagnose students' weaknesses in skills that 
are known to be important for certain occupations. For example, 
composite scores in word knowledge, paragraph comprehension, 

43. Taken together with findings of strong correlations between the Armed Services 
Vocational Aptitude Battery (ASVAB) performance and earnings, on the one hand, and 
with other studies demonstrating strong correlation between vocational program 
participation and earninci in: training-related jobs, OTA concludes that ASVAB cannot 
capture all the correlates of participation in vocational programs. Simply put, additional 
vocational education contributes to labor market success in ways that cannot be 
measured by ASVAB. For example, participation in certain vocational programs may be 
attractive to.^employers regardless of an individual's specific competencies; or students 
might learn skills in school that are not included in the test items. 



arithmetic* r'^asoningi and mechanical comprehension are used by the 
military to fill a wide- range of jobs in the "health, social, and 
technological" fields. The extent to which a school uses the ASVAB to 
diagnose individual learning needs could be an indicator of that school's 
responsiveness to broad labor market demand. Note that this use of 
the ASVAB would include reliance on its academic subtests, which 
would signal the potential value of raising the academic abilities of 
vocational students. 

Because ASVAB scores correlate strongly with "trainability," and 
because a large number of firms provide ongoing education and 
training to their employees, test results could be useful in industry. To 
be effective, however, this use of the ASVAB would require employers 
to make greater use of test results and other student information than 
they currently do. It would be particularly important for employers to 
look at grades in specific courses as well as composite test scores. 

Finally, the ASVAB could provide information tc be used in conjunction 
with other measures of performance. Early ASVAB scores, e.g., of 
incoming 10th graders, could establish baseline differences in ; ehools 
in terms of their students' prior abilities and knowledge. Such 
information could become very valuable in making later comparisons 
of earnings and employment of graduates of various schools. In 
addition, comparing early scores with scores after several years in a 
vocational curriculum could provide information on net gsins in 
knowledge and skills attributable mainly to program characteristics. 
•The' validity of' this use of ^he ASVAB has not been explored by OTA. 
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OCCUPATIONAL COMPETENCY TESHNG: STATE ACTIVITY 

According to a recent surve:^, 13 States are engaged in testing the occupational 
competencies of vocationaL and technical students and 7 States are in the process rf 
developing competency tests for vocational students. An additional 11 States are 
considering development, and 20 have indicated that they have no plans to initiate 
statewide vocational competency testing (see figure 3-2).^^ 

Many States have formed consortia to pursue the development of competency- 
based curricula and tests in a cost-effective manner^^ (see box 4, page 61). Although 
several of these consortia only produce occupational tests, some also produce materials 
tliat list the duties, tasks, and tools needed for a broad range of jobs. Often these 
materials are based on detailed job analyses conducted by small groups that include 
vocational educators? experts currently working in the job, and/or by representatives of 
associations that represent the profession with which the job is identified, in some cases, 
tests are then developed tO reflect the duties, tasks, and skills identified through these 
job analyses 'and packaged into tests or made available to States, schools, or districts in 
the form of banks of items. Educational agencies "buy into" these item banks, and can 
then select those items that match their vocational programs. Often, States pc-ohase 
the test "banks- and make them available to local education agencies, who can then 
customize the tests to match their local curricula. 

44. Rebecca Love- Wilkes, "1988 State of the Art Report on Statewide Student 
Competency Testing in Vocational and Technical Education," pn^pared for National 
Network for Curriculum Coordination in Vocational-Technical i!ducation and the 
Southeast Curriculum Coordination Center, 1988» 

45. Competeney-bciied education systems develop curricula relevant to requirements 
for working.in modern society, and attempt to certify student progress on the basis of 
demonstrated performance in some or all aspects of those requirements. Theoretically, 
such demonstrations of competence are independent of time served in formal education 
settings. See G. Grant et al. (eds.), Competence: A Critical Analysis of Competence- 
abased Reforms in Higher Education (San Francisco^ CAj Jossey-Bass Publishers, 1979), p. 

6. 
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SOURCE: QiUc4 of T«chnology Asstssmtni, based on dolo from Rtbecca Lov«-WIIk«s, 1088 Stota of th< Ai\ ft port on 
StoUwId* Student Compattncy Tastlrg In Vocotionot ond Tachnico! Educof^on (Mississippi Stole, MS: Research 
and Curriculum Unii for Vocatlonol. technical and Adult Education. t988). 
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While these types of tests crji provide more accurate information about program 
quality than tests of generic abilities, they are more time-consuming to develop; it is 
possible that by the time a test is administered, the skill it measures is obsolete. In 
addition, because they emphasize proficiency at highly specific occupational skills, these 
tests could discourage educators from exploring ways to teach broadly applicable skills 
that could benefit students in the long run. It is important to keep in mind the tradeoffs 
Inherent in the application of various types of tests. 

Occupational compevency tests assess skills (tasks) and knowledge found in specific 
jobs (e«g., changing fuel filters or preparing schematic drawings), and are different from 
tests of academic competency (e.g., solving algebra problems or writing essays), and 
tests of amployability skills (e.g., knowing how to conduct a job search, completing a job 
application, or exhibiting acceptable work habits.) 

Tests can take several forms: paper and pencil objective tests (e.g., multiple 
choice, true-false), paper and pencil subjective tests (e.g., writing sample), performance 
tests (e.g., observations of student performance of a task), portfolio assessment (e.g., 
judging a body of work, such as photographs of hair styles completed over a semester's 
work). Each of these assessment modes has strengths and weaknesses, depending on how 
they are applied. While performance testing is generally more faithful to the actual task 
as performed in a job situation, it is costly to administer. 

Ihe level of cognitive* complexity is an important characteristic of competency 
testing. Some skills or concepts are more complex than others and therefore more 
difficult to measure. At one extreme are measures of simple recall skills, such as "list 
the types of fuses and breakers." More complex measures are needed to assess an 
individual's ability to orchestrate and apply various configurations of simpler skills, as in 
the, following item from a widely.used test: "Redesign entrance service requiremeiits for 
building expansion, renovation or installation of heavy powered equipment not accounted 
for under original construction."^^ 




HOV/ GOOD ARE OCCUPATIONAL COMPETENCY TESTS? 



If occupational competency tests are used to assess the effectiveness of a 
secondary school vocational program^ it should be able to gauge the competence of 
students who are in that program* The test should also be a valid indicator of the 5kilis 
and knowledge used in the job for which the vocational program is training students. 
Obtaining estimates of '*job validity" requires input from those working in the jobs in 
question. Those who determine the composition of the test should be familiar with the 
joos as currently practiced in that region^ and should be active enough in the profession 
to be aware of trends and directions the profession is likely to take, particularly with 
respect to technological advances. 

In some instances, the composition of the committees that determine what will be 
measured on 'these tests may be too dependent on vocational teachers who may not be as 
aware of the current job tasks and responsibilities or projected changes as someone 
currently working in the profession. A test developed with the exclusive input of 
teachers may neglect modern techniques and technology, which would undermine its 
validity for predicting job placement or performance. 

One approach to determining a test's job validity is to compare the performance of 
experts — who have been j/Crforming the job successfully for several years — with that of 
novices — students beginning their training in a field or workers in a different 
occupation. Examining the ways in which experts answer specific test items gives 
important information about the validity of inferences from the responses of novices: 
for example, if an individual who is an expert automobile mechanic errs on a question 
about brakes or transmissions, that could signal the need to revise the question before 



46. From Vocational Technical Education Consortium of States (V-TECS), Vocational 
Education Program Courses Standards^ Industrial Education (Atlanta, G A; 1988). 



Including it on a test adtninistered to recent graduates. But expert-novice validity 
- checks are seldom conducted by the agencies engaged in occupational testing at the 
secondary leveL 

A longer-term approach to evaluating validity would invoKe administering the test 
to vocational completers and then comparing job placement, productivity, performance, 
and wages of those who scored well on the test with those who scored poorly, in both the 
short term and the long term. OTA finds some evidence that specific occupational 
competep^y tests correlate quite strongly with job performance. An analysis that 
synthesized the results of 262 studies of occupational competency tests^^ concluded that 
the average correlation betwf^en test scores and sutiervisbrs* ratings of job performance 
was .51 (very high). In fact, this correlation was higher than that of any other predictor 
variable ^uch as ability tests, psychomotor tests, interviews, and biographical 
inventories). Other studies have found similar- results^^ suggesting that occupational 
competency examinations appear to be valid predictors of job performance and 
promotion probabilities. 

There are a number of risks inherent in any testing effort. One cited frequently is 
the incentive to "teach to the test": if teachers prepare students by teaching them to 
answer the items on a specific test, then the test provides a measure of how well 
students have learned answers to those test items but remains ambiguous with respect to 
students* learning of the more general skills and knowledge. However^ this tendency 
should be distinguished from the case in which the use of a test is intended to influence 
instruction. For example, the New York State Department of Education has specified 
that all students must be proficient in certain occupational skills and knowledge, and has 
developed tests to measure those skills. Educators are responsible for tailoring their 

47. M.D. Dunnette, American Petroleum Institute, "Validity Study Results for Jobs 
Relevant to Petroleum Refining Industry," unpublished manuscrpt, 1972. 

48. See, for example, J.E. Hunter, "Causal Analysis, Cognitive Ability, Job Knowledge, 
Job Performance, and Supervisor Ratings," Performance hfeajure und Theory, S. Lundy et 
al .'.-^ds.) (Hillsdale, NJ: Lawrence Erlbaum, 1983). 
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- instructioa so that it coincides with^>^at is tester* In this case, poor test performance* 
by students could reflect the failure of their teachers to meet defined curricular 
objectives. In general, tests used to evaluate the effectiveness of an instructional 
program should only measure what is taught. If there is a mismatch, then either the test 
or the instructional program should be revised* 

One way to overcome the problem of teact^ing to the test (or what is also referred 
to as "measurement driven instruction") is to specify the domains of knowledge, skills, 
and concepts that are to be measured. The merits of these specifications are 
noteworthy: they allow test developers to produce a bank of consistent items that all 
measure a given skill in the same way. More important, these specifications can be given 
to teachers and curriculum developers so that they know what will be on the test without 
seeing specific test questions. However, very few test developers prepare suo' lomain 
or item specifications, which are costly and intelleelually demanding. It is not possible 
to ascertain the extent of teaching to the test that occurs in vocational programs 
without intensive case studies, and OTA was not able to determine whether the testing 
efforts reviewed for this Background Paper used item or domain specifications. 

The subtleties of testing can be easily blurred in the rush to evaluate and improve 
programs. And because tests can create such powerful incentives for change — on the 
part of students, teachers, and schools — their application warrants close attention. The 
process of developing competency tests involves substantial input from those working in 
the field. If developed properly these tests can guide vocational instruction so that it 
meets the needs of employers and students. When linked to data reflecting labor market 
needs, the tests are useful for evaluating the relevance of vocational curricula to current 
and anticipated conditions. However, OTA finds that there are too many different 
occupations in which schools are training students for there to be a viable national 
competency test. The Federal Government could effectively work together with the 




States in-developing test guidelines and^exploring new test methodologies. There is much 
to be gained from continued Federal support for research into the theory and practice of 
competency testing* 
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Box 4 

State Activity in Development of Competency Tests 



Several States and organizations are involved in 
developing and/or marketing occupational competency testing 
systems available for use in vocational education: 

National Occuf .tional Competency Testing Institute (NOCTI) 
NOCTI develops and provides occupational testing 
services for teachers, students, and industry. At the present 
time, 47 States and the District of Columbia participate in 
NOCTI, and 23 States use NOCTI tests for certifying 
vocational teachers. Both written and performance tests are 
used to assess each occupation with tests covering factual 
knowledge, technical information, understanding of 
principles? ^ problem-solving abilities. The performance 
tests are administered in laboratory, industrial, or clinical 
settings and consist of work assignments that require 2 to 5 
hours 2o complete. NOCTI test are only available for us at 
the end of a program, not for ongoing assessment during an 
instructional program. 

Costs of assessment vary, ranging from $1 per student 
in architectural design to $45 per student in construction 
masonry. At the present time, 34 Student Occupational 
. Competency Achievement Tests have been developed md 
validated. An ac'ditional 23 have been validated through field 
-testing and are-being prepared for distribution, and 11 are in 

(continued) 
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the process of: validation.' NOCTI provides scorii^ and report 
services* 

Ohio Vocational Education Achievement Test Program 

The Ohio Vocational - Assessment program is a joint 
venture between the Ohio Department of Education's Division 
of Vocational and Career Education and the Instructional 
Materials Laboratory at Ohio State University. As with the 
other testing efforts^ the Instructional Materials Laboratory 
(IML) derives test items, from an analysis of the duties and 
tasks involved in each occupation. These duty and task lists 
are developed by committees comprised of vocational 
teachers, State education staff, and a nonteacher member of 
the occupation* In addition, industry representatives 
throughout the State are surveyed for their comments on the 
task lists. Mathematics, science, and communications 
competencies relevant to the occupational competencies are 
included in the task analyses published by IML. Tests are 
comprised of two parts with approximately 175 questions for 
each part, averaging about one item for each task level. 
Performance tests are not a part of the IML testing effort. 
Each March approximately 60,000 vocational students are 
tested in the State of Ohio with scoring and report services 
provided co schools by IML. 

The tests are de£jigned for Vi2<t by teachers, supervisors, 
and administratorr. for evaluation and diagnosis of vocational 
achievement for the improvement o' instruction. Although 

(continued) 
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participation in- the testing program is voluntary, almost aE 
eligible secondery schools in Ohio are involved. 

Within the State of Ohio, the co ,t of testing per student 
is $1.50, and for schools outside of the State the cost is $2.50 
if testing-is done in-March (the time scheduled for testing in 
Ohio) and $3.50 if done at some other time. No scoring 
services are provided outside of Ohio. 



competency tests that are given to students to assess their 
mastery of selected vocational coursework. At this point in 
the development process, these tests are primarily used for 
entry level courseiJ, such as "Introduction to Occupations." 
Students must pass the tests as one of several 
requirements for high school graduation. New York's tests 
contain only multiple choice, primarily knowledge-based 
items, and do not measure students* abilities to perform job- 
related tasks. Nev; York participates in virtually all of the 
testing consortia and make,-^ these services available to 
schools and districts within the State who wish co test 
students in areas beyond those offered by the State 
competency tests. No studies of job-test validity have been 
conducted othenthan review of test items by those working in 
the field. At^-the. present time, the tests developed by NYSDE 
are not available outside of New York. 



New York State Department of Education (NYSDE) 



NYSiDL is in the process of developing occupational 



• (cofitiintxed^*^ 
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• Vocational.Technical Consortium of States (V-TECS) 

V-TECS is a consortium of 25 State education agencies 
formed to promote the systematic development and 
implementation of competency-based vocational-technical 
education.. Although most^of tht materials and test items are 
developed by S*^ate agencies, the pi^ocess is guided by 
specifications proTrided by the V-TECS staff to help ensuve 
quality control and consistency of products. 

The V-TECS materials derive from thorough analyses of 
job tasks, skills, and knowledge. These analyses are 
conaucted by actual observations of approximately 12 to 15 
individuals as they perform a job. Based on these job 
analyses, task-skill lists are developed, and reviewed by 100 
to 200 individuals who represent various aspects of the job or 
profession. The catalogs, which contain task descriptions for 
esch job are then used by States or districts to develop tsst 
items, instructional materials, and curriculum guides. 

The test items ^developed by the States are contributed 
to the V-TECS item banks from which States " schools can 
draw to comprise a test that matches the local curriculum. 
V-TECS does not currently provide intact tests or scoring 
services, but is considering moving to centralized test 
development, in part to ensure more quality control. V-TECS 
currently -has catalogs (job djsciiptions) for 180 job titles, 
with one-third of those titles being updated at any one time, 
n addition, 12 to 15 test item banks have been completed and 
another 10 are nearing completion. Each item bank contains 



(continued) 
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appFOximately 500 to 1,500 Jtems for a job area, with 3 to 20 
items for each skill, -^any of these are tests of actual 
performance. Having this breadth in the number of test 
items offers some degree of test security and permits 
.... flexibility. in generating.multiple forms of a^est. V^TECS is 
also in the process of identifying basic academic skills for 
infusion into their occupational materials, but have not yet 
addressed higher-order thinking or problem-solving skf'\% 

The costs of participation in V-TECS is $20,000 por 
State per year. As part of their agreement, each State 
agrees to produce one (small States) or two (larger States) 
V-TECS products. 

Oklaboma State Department of Vocational and Technical 
Education 

Oklahoma offers one of the most advanced competency- 
based curriculum and testing systems in the country. They 
have been leaders in a competency-based curriculum for over 
25 years and in 1983, began a systematic and comprehensive 
effort in occupational testing. At the present time, they 
have completed development of test batteries for 168 
occupations, each of which consists of three types of tests. 
The first is the traditional multiple choice (cognitive) test, 
the second is a scenario that is designed to assess students' 
decisionmaking skills within the occupation, and the third is a 
performance test. In addition, 600 to 1,000 multiple choice 
test items have been developed for each occupational test 
b-^ttery. This permits the composition of a variety of (continued) 
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randomly .generated tests drawn from the larger battery of 
test items- that measure a single domain. 

All test items are keyed to duties and tasks that have 
been identified by committees of industry and educational 
representatives. Approximately four to six individuals 
representing each group sit on a committee. There is a great 
deal of concern with ensuring that the duties, tasks, and test 
items represent the most recent state of the industry. 

Oklahoma does not require use of these tests, but makes 
them available to local programs for use in evaluating 
students' mastery of competencies and for identifying 
strengths and weaknesses in curriculum and instruction. At 
present, approximately 60 percent of the schools in Oklahoma 
have used the .tests in some way. Apparently there is some 
concern on the part of teachers that they will be evaluated on 
the basis of their students' performance, although it is the 
teacher who requests the test from the State and who 
receives the test results. 

The cost of the duty-task lists are $2 to $4 for in-state 
teachers and for out^-of state, the cost is $6 per occupation 
contained in a book, which may range from 3 to 14 
occupations per book The cost of a test for out-of-state 
purchasers is $30 and contains 20 student packages, and 
includes all three types of tests mentioned above. Oklahoma 
•does not provide scoring or reporting services for those 

(continued) 
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outside of -the - State. Approximately 95 pprcenC: of the 
occupations - .taught, in secondary vocational programs are 
covered by the Oklahoma system. 
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Chapter 4 
Other Performance Measures 

Vocational education at the secondary level has many objectives in addition to job 
preparation and job placement!*^ Three ..Jrticularly important areas not examined by 
OTA are discussed in this section; academic skills, higher-order thinking skills, and 
postsecondary educational attainment* 

ACADEMIC SKILLS 

Many recent reports have called for elevating the basic academic skills of all 
students* The National Governors' Association, for example, argued that the task of 
vocational education Is to prepare . . workers who are both well educated with the 
ability to reason and adapt, and well trained to perform specific work tasks."^^ (See box 
5, page 75*) 

Forty-two States responding to OTA's telephone survey reported efforts to 
introduce academic material into th^ vocational curriculum. As shown in figure ^.-1, 32 
States are In a consortium that is devCfloping special ccurses in applied r^\th.^matics, 
science, and communications.^^ Two States currently require the assessment of 
vocational studentr* academic abilities, and a consortium of southern States has been 
formed to advance the development of basic competencies of students enrolled in 
vocational programs (see box 6, page 77). 



49. For a discussion of noneconomic effects of vocational education, see Lawrence 
Hotchkiss and Linda Dorsten, "Effects of Tracking on Post-High School Outcomes," 
Sociology of Education and Socialization^ vol. 6, Ronald Corwin (odO (Greenwich, CT: 
JAI Press, 1987), pp. 191-219. 

50. National Governors* Association, "Draft Policy Statement on Vocational 
Education," unpublished manuscript, 1988. 

51. This development work is being conducted through a consortium cf State agencies 
called the Center for Occupational Research and Development (CORD), in Waco, TX. 




Figure 4'"1 

Integration of Academic Material into Vocational Education Curricula 
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States using CORD curriculum moterials^ 
states using SREB curriculum materials 



States usmg CORD and SREB curriculum materials 
States using ttieir own curriculum materials 



^ Center for Occupational Researcti and Development (CORD) consortium 

b Souttiern Regional Education Board (SREB) consortium SOURCE; OTA teleptione survey, January 1989. 
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ERIC 



Many. analyses point.to a longrterm shift toward higher-skill jobs. It is important 
to note that«-while the rate of growth in jobs that require more advanced skills is quite 
high, the majority of the total number of jobs will continue to require relatively lov; 
skills. Compared to workers who have good academic skills, workers who are deficient in 
basic skills will experience longer durations of unemployment and will have greater 
difficulty finding new jobs at their previous wage levels. 

Another factor contributing to the demand for a better educated work force is the 
restructuring of jobs. Many jobs that were once quite simple have become more 
complex.^^ In the insurance industry, for example,.a single claims adjuster now dc3s the 
work of five. The computer has eliminated many jobs, and the customer assistance clerk, 
claims adjuster, file clerk, messenger, a * policy writer have been r >llapsed into a single 
position that demands advanced levels of communication and thought. The adjuster must 
be able to analyze a customer's needs and interpret and use the multiple arrays of 
information now available through computerized databases. Insurance companies that 
formerly hir^d high school dropouts or graduates to fill the jobs of messenger and file 
clerk now hire individuals with at least 2 years of college. 

As technology changed, so has the management philosophy of some 
organizations. For example, team work has become one strategy to handle greater 



52. See, for example. Sue E. Berryman, "The Economy and American High Schools: 
What Should We Teach? When? How? To Whom?" presented at the Tenth Annual 
Research Conferanee, Association for Public Policy Analysis and Management, Seattle, 
WA, Oct. 27-29, 1988. 

53. M; Podgursky, "Job Displacement and Labor Market Adjustment: Evidence From 
the Displaced Worker Survey," prepared for National Academy of Sciences, Panel on 
Technologjr and Employment, 1987; P-0. Flaim and E. Sehgal, "Displaced Workers of 
1979-83: How Well Have They Fared?" Monthly Labor Review, vol. 108, No. 6, pp. 3-16. 

54. Thomas^ Bailey, National Center on Education and Employment, Teachers College, 
Columbia University, "Education and the Transformation of Markets and Technology in 
the Textile Industry," Technical Paper No, 2, 1988; Thomas Bailey and Thierry Noyellep 
National Center oh Education and Employment^ Teachers College. Columbia University, 
"New Technolpgy and Skill Formation: Issues and Hypotheses," Technical Paper No. 1, 
198S; Thierry Noydie, Beyond ^dustrial Dualism {Boulder, COt Westview Press, 1387); 
and Thierry Noyelle, National Center on Education and Employment,. Teachers College, 
Columbia University, "Services and the New Economy: Toward a New Labor Market 
Segmentation," Occasional Paper No. 5, 1988. 



70 

Si 



workplace complexity} hut team work requires skills in problem identification, conflict 
resolution, and project:evaluation t^at many workers- may need to acquire. 

For these rceHors, many people advocate the inclusion of basic academ* 2 
competencies in performance measurement. However, there are also many people who 
believe that the primary, t^oal of vocational education should be mastery of occupational 
skills. Clearly there is some overlap. The problem is how to fine-tune the blend of job- 
specific and generic skills in a curriculum that may already be quite dense. While some 
educators and administrators believe that schools can accommodate both, there is also 
some evidence thai too much emphasis on academic content could undermine the 
vocational curriculum and leave vocational students worse off thai^ before.^ 

There are other reasons why OTA urges cautic ^n the implementation of stringent 
academic require'. icnts. First, many of the students enrolled in vocational education 
programs are there piaeisely because they were not successful in more academically- 
oriented classrooms. Adding more rigorous academic material to the vocational 
curriculum could frighten off the very students for whom vocational education is 
Guceessful. Second, proponents of a more occupationally*oriented vocational curriculum 
point out that vocational graduates caa support themselves through college thanks to the 
skills they acquired in h^gh school. Since college-going is increasingly seen as an 
objective for many students who complete vocational programs in high school, it would 
be a mistake to vceaken these students' technical occupational credentials. Third, there 



55. In fact, when management ap.rees to funilamental changes in the decisionmaking 
role of employees, productivity can improve even among a work force whose skill level 
was originally considered inadequate. See, for example, Murnane, op. cit., footnote 1, p. 
218; Richard M. Cyert and David C. Mowery (eds.). Technology and Employment: 
Innoyztion and Grov;th in the US. Econo^'iyX Washington, D^s National Academy Press, 
1987); and Berryman, op., cit., footnote 53. . 

56. See, for example^ William Clune et al., Center for Policy Research in Education, 
Eagleton Instituti of Politics, Rutgers. University, "The Implementation and Effects of 
High School G^'^duation , Requirements: First Step? Toward Curricular Reform," 
unpublished manuscript, 1989^ In an article in Education Weak (Feb. 15, 1989, p. 5), 
Clurie is quoted as saying that . . vocational courses were the major casualty of the 
'increased-academiccoursetaking. . • « We saw evidence thac the new requirements made 
it difficult to complete logical, and even required, sequences of vocational courses." 



71 85 



are reasons to. be, skeptical about some projections of skill requirements, as well as about 



statistical correlatioa«* between performance on tests of cognitive ability and *ob 

err 

performance. 

HIGHER-ORDER THINKING 

Many educators have come to believe that knowing how tc learn nas become as 

CO 

i n.portant as whal learned . As one researcher has argued: "We are moving into an 
Bi^ in which the trac tional separation between working and learning is disappearing^ 
with learning becoming increasingly integrated into a person's work life." Our aim, 
claims another researcher, should be to develop "expert novices" v;ho, ". . . although they 
may not possess sufficient background knowledge in a new field, know how to go about 



gaining that knowledge."'' The term most commonly associated ;ith reasoning and 
problem solving is "higher-order thinking skills." This is an abstract term, difficult to 
define and even more difficult to observe.^" But it is relevant to performance 



5*^. For a mo?e thorough treatment of these issues see Murnane, op. 6it., footnote 1, 
pp. 218-225. Murnane points out that while certain cognitive skills are important for all 
students, it is not entirely clear how high their skill levels must be: ". . . the best 
available information indicates thLt enhancing productivity growth will require that all 
students be provided with threshold levels of literacy and problem-solving skills, as 
measured on paper-and-pencil tests. Extremely high scores on the types of standardized 
tests typically used in schools to measure cognitive skills may not be necessary fot 
productive performance in the labor foree." (p. 224) 

58. This attitude, too, is controversial. See E^D. Hirsch, "The Primal Scene of 
Education/* New Vorfc Review of Booto, Mar. 2, 1989, pp. 29-34. 

53. See Noyelle, iSeyond Industrial Dualism ^ op. cit., footnote 55; Robert Glaser, 
"Teaching Expert Novices," Educgttional Researcher (Washington, DC: American 
Educational Research Association, December 1987); and Berryman, op. cit., footnote 53, 
who lists the following com,jonents of knowing how to learn: ". . . knowing how to 
identify the limits of one's own knowledge, how to ask germane questions, how t( 
penetrate poor documentation, and how to. identify sources of mformation." 
60. One researcher suggests that higher-order cognitive skills are those that involve 
the crchestration ^and^ practical use of the simpler skills — lik^ computation in school 
' arithrietic. See Susan F. .Chipman, V^hat is Meant by ^Hi£herOrder Cognitive Sifdlls" 
(Arlington, VA; Office of Naval Research, 1987). 

Another .definition uses the following characterizations of higher-order thinking: 
nonalgorithmic,v meaning that the path of action is not fully specified in advance; 
complex, meaning that the total path is not mentally "visible" from any single vantage 
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measurement in vocational education because of the interest in moving beyond iimple 

measures^olshort-run job placement and toward providing students with . • a job with a 

future — a job that provides personal growth, the chance to master new skills, and the 

fit 

opportunity to earn promotions." To the extent that t'lese higher skills cm be 
diagnosed and measured, they could become. elements in a comprehensive syste.Ti of 
pei^formance measurement. At present, however, testing methods have not reached tha 
level of sophistication, wruch suggests that the Federal role should be restricted to 
support for basic research.^ 



EDUCATIONAL ATTAINMENT 



Apart from what students learn in school, how far they go in school has important 

go 

consequences of its own . Finishing high school has significant advantages: between 
1961 and 1981, the average earnings of full-time working men (aged 25 to 64) with 1 to 3 
years of high school fell from 87 to 75 percent of the average earnings of a comparable 
group with 4 years of high school. Thus, it is understandable why reducing the high 
school dropout rate is frequently mencioned as an important objective of vocational 



point; often yielding multiple solutions, each with costs and benefits, rather than unique 
solutions; involving nuancod judgment and interpretation; requiring the application of 
multiple, sometimes conflicting, criteria; involving uncertainty — not everything bearing 
on the task is known; involving self-regulation of the thinking process, not regulation by 
otliers; involi '•"^ imposing meaning, finding structure in apparent disorder; and being 
effortful." Sue Luuren Resnick, Education and Learning to Think (Washington, DC: 
National Academy Press,. 1987), p. 3. 

The limited current base of knowledge about higher-order coi^piition is discussed in 
a recent OTA Technical Memorandum on elementary and secondary schooling for future 
scientists and engineers. See U.S. Congress, Office of Technology Assessment, 
Elementary and Secondary Education for Science and Engineering, OTA-TM-SET-41 
(Washington, DC: U.S. Government Printing Office, December 1988), pp. 77-81. 
Gl.- The Wiaiam T. Grant Commission-.on Work. Family, and Citizenship, The Forgotten 
Half: Pathways to Success for Americans Youth and Young Families (Washington, DC: 
1938). 

62. Four States are already either plamiing- or implementing the assessment of higher- 
crder thinking and. probl^mrsolving skills. 

63. This discussion draws on Steru, op. cit«, footnote :5, pp. 10-12. 
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education. Indeed^the fact' that so many students elect to take vocational courses in 
..high, school means^ thai' some students would have less reason to come to school if those 
courses were not available. While statistical studies of the effect of vocational 
coursework on dropout rates have been ambii^uous^ continued research in this area could 
be .fruitful -and might yield-, new quantitative measures program performance. 
Similarly, the propensity for vocational students to attend college — again, independently 
of their long-run labor market outcomes — warrants farther study. 
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Box 5 

The Need.for Basic Skills: Mounting Pressure^^ 



•Future-jobs will be restructured about-every seven years 
and work and learning will be inseparable. 

-David Kearns, Chief 
Executive Officer of 
Xerox Corp. 



More than half of all new jobs created between 1984 — 
2000 will require some education beyond high school, and 
almost a third will be filled by collef^' graduates. Today, only 
22 percerit of all occupations require ^ college degree. 

-Bureau of Labor 
i.tatistics, U.S. 
Department of Labor 



The pace of technology development now is so greet 
that life cycles for electro:iics products and processes already 
have collapsed to three to five years, and rarely will exceed 
five to ten years in most other industries. As a result, any 
set of skills also can be obsolescent in five to ten years. 
Continuous re-skilling must he a top national priority. 

-U.S. Department of 
Commerce 



Tektronix, an Oregon based manufacturer of electronic 
equipment, tried to shift its traditional assembly line 
workforce to a flexible manufacturing system four years 
ago. The company discovered that 20 percent of its 
production.workers lacked rudimentary skills ^--eeded tc. the 
transition. Tektronix is solving its problem by contracting 
with nearby Portland Community College to tun a remedial 
on-site program in basic math and English for its many non- 
English speaking assemblers. Along the way, Tektronix added 
courses to enhance such skills as team building, negotiating, 
and effective time management. 

-Fortune Magazine , 
Apr. 11, 1988 



64. The following text is quoted from U,S. Department of 
Education^ U.S. Department of Commerce, and U.S. 
Department of Labor, Building a Quality Workforce 
(Washington, DC: U.S. Department of Labor, July 1988 . (continued) 
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Research done at Indiana University has found that 
some 70:percent of the reading material In a cross section of 
jobs nationally is now between 9th grade and 12th grade 
difficulty (some 15 percent is even higher) — and it is likely 
that the job and social roqulrements for literacy will increase 
even more in the years ahead* 

-Business Council for 
Effective Literacy 
Newsletter , April 1988 



Experts both in and out of government agree that the 
competitive global marketplace, increased domestic 
competition diie to deregulation, the pace of technolpgy 
development, shorter pioduct life cycles, and new flexible 
production processes all require more ad^^able, more highly 
educated entry workers in order for our country, to remain 
competitive* 

-Building a Quality 
Workforce , U.S. 
Department of 
Education, U.S. 
Department of 
Commerce, and U.S. 
Department of Labor, 
1988 
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Box 6 

. Academic Material in the Vocational Curriculura: 
State Efforts 

The Southern Regional Education Board (SREB) is a 
consortf.um of 13 States working toward introducing academic 
mathernatics and science material into the vocational 
education curriculum. ^ The stated purpose ^ ' SREB is to; 

* « advance, ^pply^ and evaluate approaches that will 
strengthen the development of the basic competencies — 
communication, science, mathematics, critical thinking, and 
problem solving — of students enrolled in vocational" 
programs." 

Educators at 33 sites in consortium States are 
inplementing a variety of strategies including: 

• having vocational students take higher 
level communications, mathematics, and 
science courses; 

• increasing the amount of emphasis and 
instructional time in vocational courses 



65. The participating States ^vei Alabama, Arkansas, 
Florida, Georgia, Kentucky, ' Maryland, Mississippi, North 
Carolina, Oklahoma, South Carolina, Tennessee, iHrginia, and 
West Virginia. 

66. James E. Bottoms, "The Relationship Between 
Vocational .-and Academic . Education," prepared .for Policy 

Studies Associates, Inc., Washington, DC, November 1988. (continued) 
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devoted -to basic competencies that 
underlie. the occupational field. of study; 



providing remediation to studenis in need; 

« 

providing staff development to vocational 
teachers on how to incorporate instruction 
in basic academic subjects. 

An evaluation plan draws information from multiple 
indicators of program effects, including results from the 
National Assessment of Educational Progress (NAEP) tests in 
mathematics, science, and reading; transcript analysis; 
student opinions; completer and employer followup 
information; and classroom observations designed to measure 
the emphasis being placed on academic competencies. 
Correlations will be conducted to determine the relat! inships 
among vocational education experience (e.g., curriculum, 
instruction, counseling), NAEP scores, and students' post-high 
school employment. Student test scores will be compared 
both with other students of similar background (e.g., sex, 

race, type of vocational program, parent education), and also | 
with the national NAEP sample. An added feature is that this I 
system y ovides the ability to analyze changes over time. 
-NAEP,.was administered to a group of- students in May 1988 
and analysis of the results is under way. 

(continued) > 
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Preliminary, analyses conducted by the Ohio State 
University and. reported in an memorandum to SREB 
members^ reveal that: 



• • • completing a general mathematics course 
added about three points to a student's NAEP 
mathematics score. On the other hand^ 
completing a pre-algebra course resulted in a 9- 
point increase; Algebra a 12-point increase; 
and geometry, a 14 point increase. Further, the 
average NAEP mathematics score for students 
who said' that their vocational teachers often 
stressed mathematics was 9 points higher than 
the average score of students who said that their 
vocational teachers seldom or never stressed 
mathematics. 

The NAEP results also provide some insight 
for improving the science achievement of 
vocational completers. For example, students in 
pilot sites that provided vocational completers 
with lab-based science courses scored 
approxinatcly 20 points higher than those 
students from pilot sites where science courses 
were prednminantly textbook — and information 
— centered. ' 



The director of SREB recommends that NAEP be used 
nationwide to assess the academic competence of vocational 
students. However, he does not see the need to use the 
results of NAEP to determine funding at the Federal level, 
since he believes that the impact of publicizing the test 
results at the State and local levels will provide the needed 



incentives for improvement. 



67. James E. Bottoms, memorandum to State ^nd Local 
Pilot Site Coordinators of the Southern Regional Education 
Boe^'d, Jan. 10, 1989. 



7:* 



93 



RECENT OTA REPORTS ON RELATED SUBJECTS 



Available from the Scieiicey Education, and Transportation 

Program, Office oi Technology Assessment, U.S. Congress, 
Washii^ton, DC 20510-8025 (202/228-6920) 

' - Demographic Trerds and the Scientific and Engineering Work Force 
(Technical Memorandum), December 1985 



Available from the Superintendent of Documents, 

Government Printing Office, Washington, DC 20402-9325 
202/783-3238 

Power On! New Tools for Teaching and Learning , September 1988 
GPO stocic number 052-003-01125-5; price $11.00 

Educating Scientists arid Engineers; Grade School to Grad School , June 1988 
GPO stocic number 052-003-01110-7; price $6.00 

Elementary and Secondary Education for Science and Engineering 
(Technical Memorandum), December 1988 
GPO stocic number 052-003-01141-7; price $7.00 

Higher Education for Science and Engineering (Background Paper), March 1989 
GPO stocic number 052-003-01148-4; price $12.00 
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